4 Ağustos 2021 Çarşamba

Subquery Expressions - EXISTS

Giriş
Subquery sadece satır olup olmadığıyla ilgilenir. Satırın içeriği önemli değildir. Bu yüzden SELECT 1 kullanılabilir.

NOT IN vs NOT EXISTS
Açıklaması şöyle
NOT EXISTS will usually outperform “NOT IN” by a good margin.

Kullanım
Örnek
bar.geom noktasından 10 birimden fazla uzak olanları foo'ları seçmek için şöyle yaparız. bar.geom ve foo.geom aynı birimden olmalı. Eğer bu ikisinin birimi metre ise 10 birim de aslında 10 metre anlamına gelir.
SELECT *
FROM   foo
WHERE  NOT EXISTS (
  SELECT 1
  FROM   bar
  WHERE  ST_DWithin(bar.geom, foo.geom, 10)
);


PostGIS ST_Dwithin

Giriş
Belirtilen noktanın X birim uzağına düşen noktaları verir

Örnek
bar.geom noktasından 10 birimden fazla uzak olanları foo'ları seçmek için şöyle yaparız. bar.geom ve foo.geom aynı birimden olmalı. Eğer bu ikisinin birimi metre ise 10 birim de aslında 10 metre anlamına gelir.
SELECT *
FROM   foo
WHERE  NOT EXISTS (
  SELECT 1
  FROM   bar
  WHERE  ST_DWithin(bar.geom, foo.geom, 10)
);

2 Ağustos 2021 Pazartesi

High Availability

Giriş

Sanırım 3 tane temel senaryo var. Bunlar şöyle
1. Running PostgreSQL Outside of Kubernetes
Şeklen şöyle

Açıklaması şöyle
The PostgreSQL cluster consists of a Master Node and a Standby Node. In case the Master Node failed for some reason (e.g., hardware or network defect) the standby server can overtake the role of the master. The PG-Bouncer in this picture is a component from Postgres and acts as a kind of reverse proxy server. In case of a failure, the switch from the Master to the Standby node can be done by an administrator or can be automated by scripts. From the view of a client, this switch is transparent.
Açıklaması şöyle. Master ve replica arasında senkronizasyon kopsa bile Postgred replica devralabilir. Bir miktar veri kaybı olsa bile
PostgreSQL'de otomatik failover'ı açmak için replikasyonu senkron moda almak zorunda değilsiniz. Asenkron bir yapıda bile, primary çöktüğünde bir replica otomatik olarak terfi edebilir. Yani "biraz veri kaybı olabilir ama sistem kendi kendine ayağa kalksın" senaryosuna izin verir.

SQL Server tarafında ise durum farklı. Always On'da otomatik failover'ın devreye girebilmesi için replica'nın senkron modda olması ŞART. Asenkron bir replica otomatik failover hedefi olamaz. Siz veri kaybını göze alsanız bile sistem buna kendiliğinden izin vermez; ancak force allow data loss diyerek, bilinçli ve manuel bir müdahaleyle failover yapabilirsiniz.

Aradaki fark aslında bir tasarım felsefesi farkı. SQL Server "otomatik bir karar asla veri kaybına yol açmamalı, veri kaybı ancak insanın bilerek onayıyla olur" diyor. PostgreSQL ise "kayıp riskini kabul edip etmemek senin kararın, otomasyonu buna göre kur" diyor. İkisi de savunulabilir, ama aynı problemi farklı yerden çözüyorlar.
2. Running PostgreSQL Inside of Kubernetes
Örnek
Deployment şöyledir
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: spring-keycloak-demo
spec:
  selector:
    matchLabels:
      app: postgres
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:latest
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: postgres
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: spring-keycloak-secrets
                  key: postgres-pass
service şöyledir
apiVersion: v1
kind: Service
metadata:
  name: postgres-service
  namespace: spring-keycloak-demo
  labels:
    app: postgres
spec:
  selector:
    app: postgres
  ports:
    - port: 5432
  type: NodePort
Örnek
Şöyle yaparız. storageClassName is specific to Kubernetes cluster
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  labels:
    component: postgres
spec:
  selector:
    matchLabels:
      component: postgres
  serviceName: postgres
  template:
    metadata:
      labels:
        component: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:11
          ports:
            - containerPort: 5432
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgres-data
          env:
            - name: POSTGRES_DB
              value: postgres
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              value: postgres
  volumeClaimTemplates:
    - metadata:
        name: postgres-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: hostpath
        resources:
          requests:
            storage: 5Gi
Service olarak şöyle yaparız
apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    component: postgres
spec:
  selector:
    component: postgres
  ports:
    - port: 5432
3. Running PostgreSQL on a Distributed Block Storage


Select For Share - Read Lock

Giriş
Açıklaması şöyle
A weaker form of select for update is the select for share query. It is an ideal for ensuring referential integrity when creating child records for a parent.
Örnek
Açıklaması şöyle
Suppose that we want to create a new purchase for a user. First, we would select the user from the database and then insert a new record in the purchases database. Can we safely insert a new purchase into the database? With a regular select statement we can’t. Other processes could delete the user in the moments between selecting the user and inserting the purchase.

One way to avoid potential issues is to query for the user with the FOR SHARE locking clause.
Şöyle yaparız
process A: BEGIN;
process A: SELECT * FROM users WHERE id = 1 FOR SHARE;
process B: DELETE FROM users WHERE id = 1;
-- process B blocks and must wait for process A to finish

process A: INSERT INTO purchases (id, user_id) VALUES (1, 1);
process A: COMMIT;
-- process B now unblocks and deletes the user
Açıklaması şöyle
Select for share prevented other processes from deleting the user, but does not prevent concurrent processes from selecting users. This is the major difference between select for share and select for update.

The select for share prevents updates and deletes of rows, but doesn’t prevent other processes from acquiring a select for share. On the other hand, select for update also blocks updates and deletes, but it also prevents other processes from acquiring a select for update lock.

Select For Update Skip Locked

Giriş
Açıklaması şöyle. Yani bir bir transaction tarafından "select for update" ile kilitlenmiş satırları hariç bırakır ve geri kalan satırları verir
Processing Non-Locked Database Rows
Select for update can be a rigid lock on your table. Concurrent processes can be blocked and starved out. Waiting is the slowest form of concurrent processing. If only one CPU can be active at a time, it is pointless to scale your servers. For this purpose, in PostgreSQL there is a mechanism for selecting only rows that are not locked.

The select ... for update skip locked is a statement that allows you to query rows that have no locks. Let’s observe the following scenario to grasp its use case:
Ne İçin Kullanılır?
Açıklaması şöyle
The SKIP LOCKED clause allows a query to skip rows that are currently locked by other transactions. This is useful for job queues where multiple workers need to process jobs concurrently without conflicts.

Örnek
Şöyle yaparız
//session 1
BEGIN;
SELECT * FROM student WHERE id=1 FOR UPDATE

//session 2
SELECT * FROM student FOR UPDATE SKIP LOCKED;
Örnek
Şöyle yaparız
process A: SELECT * FROM purchases
process A:   WHERE processed = false FOR UPDATE SKIP LOCKED;
process B: SELECT * FROM purchases
process B:   WHERE created_at < now()::date - interval '1w';
process B:   FOR UPDATE SKIP LOCKED;
-- process A selects and locks all unprocess rows
-- process B selects all non locked purchases older than a week

process A: UPDATE purchases SET ...;
process B: UPDATE purchases SET ...;
Örnek
Şöyle yaparız. Burada aynı anda birden fazla kişi bu cümleyi çalıştırsa bile, birbirlerini beklemedikleri için aslında bir anlamda işleri bölüşerek çalıştırma imkanı oluyor. Mesela 100 tane satır varsa ve Limit 10 ise, yani 10'luk batch'ler halinde çalıştırıyorsak, 4 kişi paralel çalışabilir.
UPDATE scheduled_tasks st1 SET picked = true, …
WHERE <instance> IN (
    SELECT <instance> FROM scheduled_tasks st2
    WHERE <due-condition>
    FOR UPDATE SKIP LOCKED
    LIMIT <limit>)
RETURNING st1.*









Select For Update No Wait

Açıklaması şöyle
Non-blocking Select for Update Statements
When the applications selects some rows for update, other processes are forced to wait for the transaction to end before they can get a hold of that lock.

If the processing takes too long to complete, for whatever reason, other parts of the system might be blocked. This can be undesirable. We can use the select ... for update nowait statement to prevent blocking calls to our database. This query will error out if the rows are not available for selection.
Örnek 
Şöyle yaparız
process A: SELECT * FROM purchases WHERE processed = false;
--- process B tries to select the data, but fails
process B: SELECT * FROM purchases FOR UPDATE NOWAIT;
process B: ERROR could not obtain lock on row in relation "purchases"
process A: UPDATE purchases SET ...;

Select For Update

"Select For Update" vs Select
Açıklaması şöyle
The difference between select and select for update is that select for update locks the rows returned as a response of the query. This means no other transaction can make any changes to those rows till the transaction holding the lock commits/abort.
Burada önemli olan şey tüm transaction'ların "Select For Update" kullanması. Açıklaması şöyle
Both transactions should use the FOR UPDATE locking. If the first transaction doesn’t acquire the write locks, the lost update can still happen.
"Select For Update" vs Repeatable Read
Açıklaması şöyle
When you read a record under Repeatable Read, you get a read-lock, but other transactions can also get a read lock, which might prevent you from making an update later. Using FOR UPDATE informs any other transactions which request a read lock that they should wait until you're finished updating the record.
Eğer Repeatable Read kullanırsak ve iki transaction update işlemi çalıştırırsa, hata olarak şunu alırız.
pq: could not serialize access due to concurrent update
Ama aslında amacımız bir transaction'ın başarısız olması değil, sadece "Lost Update" problemi olmaması. 
"Select For Update" Uses Row Share Lock
Açıklaması şöyle
The select ... for update acquires a ROW SHARE LOCK on a table. This lock conflicts with the EXCLUSIVE lock needed for an update statement, and prevents any changes that could happen concurrently.
Örnek
Bir örnek şöyle
Case 1: Consider you are on Book My Show trying to book tickets for your next movie. You and Mr X selected the same seats for a show and proceeded for checkout. Ideally only one of you should be able to book your ticket. However, is it possible that both of you are able to book the exact same seat ?
Hatalı kod şöyle
## Psuedo code for booking 
if (seatsAvailable(List<Seats> selectedSeats)) {
    bookTickets(selectedSeats);
    sendEmailNotification(customer_id);
}
// These are two individual transactions from db point of view
seatsAvailable() {
  Select booked from booking where seats = 'H3';
  return !booked;
}
bookTickets() {
  update booking set customer_id = '123' and booked = 't' where
  seats = 'H3';
}
Doğru kod için şöyle yaparız
// This is a single transaction from db point of view
begin;
seatsAvailable()-> Select * from booking where seats = 'H3' FOR UPDATE;
bookTickets()-> update booking set customer_id = '123' and booked = 't' where seats = 'H3';
commit;
Eğer atomic yapmak istersek şöyle yaparız
update booking set user_id = '123' and booked = 't' where seats = 'H3' and booked = 'f';
Örnek
Bir örnek şöyle
Case 2: Consider a case where you are trying to purchase goods that are worth more than your current e-wallet balance. Is it possible to successfully complete the transaction thereby making your balance negative ?
Örnek
master_counter tablosundan bir değer çekip, daha sonra payment_code tablosuna bir kayıt eklemek isteyelim. İşimiz bitince de master_counter tablosunu güncelleyelim. Şöyle yaparız.
BEGIN;
SELECT counter FROM master_counter FOR UPDATE; // notice this
// Do Addition to counter in application
// Apply Generator Logic in Application
INSERT INTO payment_code(payment_code) VALUES(generated_value_with_counter);
UPDATE master_counter SET counter=<new_value_from_application>;
COMMIT;
Örnek - Spring
Elimizde şöyle bir kod olsun
@Query(value = "select * from products where id = ?1 for update", nativeQuery = true)
Optional<Product> findByIdWithWriteLock(Long productId);
Şöyle kullanalım
@Transactional
public Optional<Order> placeOrder(Long productId, Long userId) {
  Optional<Product> product = productRepository.findByIdWithWriteLock(productId);
  product.orElseThrow(() -> new RuntimeException("Invalid product id"));

  if (product.get().getAvailableUnits() > 0) {
    productRepository.decrementAvailableUnitsCountBy1(productId);
    Order newOrder = new Order(userId, productId);
    orderRepository.save(newOrder);
    return Optional.of(newOrder);
  }
  return Optional.empty();
}