JSON Indexing

Giriş

Açıklaması şöyle

PostgreSQL offers two types of indexes to work with JSON data.

- B-Tree index

- GIN index/Full-text search index.

1. B-Tree Index

Örnek

Index olmadan bir sorgu çalıştıralım

EXPLAIN ANALYZE SELECT * FROM books_data WHERE data ->> 'author' = 'Jessica Evans';

Çıktı olarak şunu alırız. Burada Parallel Seq Scan kullanıldığı görülüypr

Gather  (cost=1000.00..36481.22 rows=3864 width=279) ...
   Workers Planned: 2
   Workers Launched: 2
   ->  Parallel Seq Scan on books_data  ...
         Filter: ((data ->> 'author'::text) = 'Jessica Evans'::text)
         Rows Removed by Filter: 257582
 Planning Time: 0.087 ms
 Execution Time: 127.673 ms
(8 rows)

Bir index oluşturalım

CREATE INDEX author_index ON books_data ((data ->> 'author')); 

Çıktı olarak şunu alırız. Burada Index Scan kullanıldığı görülüyor

Index Scan using author_index on books_data  ...
   Index Cond: ((data ->> 'author'::text) = 'Jessica Evans'::text)
 Planning Time: 0.298 ms
 Execution Time: 0.091 ms
(4 rows)

2. GIN index/Full-text search index.

GIN yazısına taşıdım

JSON Path - XPath Benzeri Bir Arama Dili

Giriş

Açıklaması şöyle

JSON Path is a powerful tool for searching and manipulating a JSON object in SQL using JavaScript-like syntax:

- Dot (.) is used for member access

- Square brackets ("[]") are used for array access

- SQL/JSON arrays are 0-indexed, unlike regular SQL arrays that start from 1

Açıklaması şöyle

The jsonpath type implements support for the JSONPath standard in PostgreSQL to efficiently query JSON data. It provides similar functionality to XPath for XML. It’s a very powerful language, and allows you to traverse json structures level by level and retrieve arbitrary sub-items using multiple assessor operators and nested filter expressions. You can have strict or lax error handling. If lax, the default, then the path engine adapts the queried data to the path, otherwise with strict, an exception occurs. 

Built-In Functions

1. size

Açıklaması şöyle

JSONPath also includes powerful built-in functions like size() to find the length of arrays. 

Örnek

Elimizde şöyle bir tablo olsun

CREATE TABLE LunchOrders(student_id INT, orders JSONB); 

Bu tabloya veri girelim. Json verisi tek tırnak içindedir.

INSERT INTO LunchOrders VALUES(100, '{
  "order_date": "2020-12-11",
  "order_details": {
    "cost": 4.25,
    "entree": ["pizza"],
    "sides": ["apple", "fries"],
    "snacks": ["chips"]}
  }'      
);

INSERT INTO LunchOrders VALUES(100, '{
  "order_date": "2020-12-12",
  "order_details": {
    "cost": 4.89,
    "entree": ["hamburger"],
    "sides": ["apple", "salad"],
    "snacks": ["cookie"]}
  }'      
);

Şöyle yaparız

SELECT * FROM lunchorders WHERE orders @@ '$.order_details.snacks.size() > 0';

2. Comparison Without Type Casting

JSON Path ile şöyle yaparız

SELECT * FROM lunchorders WHERE orders @@ '$.order_details.cost > 4.50';

JSON Operators kullansaydık şöyle yapmak gerekirdi. ->> ile Filtering Query kullanılıyor

SELECT * FROM lunchorders WHERE  (orders -> 'order_details' ->> 'cost')::numeric > 4.50;

NOT NULL Constraint

Giriş

Not null kısıtı create table esnasında veya daha sonra alter table ile eklenebilir.

Örnek

Şöyle yaparız

create table <name> (
  ... columns ...
  constraint either_email check (email is not null or p2pmail is not null)
);

Örnek - alter table

Elimizde iki tane sütun olsun. Herhangi birisinin dolu olduğunu garanti etmek istersek şöyle yaparız. Böylece email veya p2pmail sütunlarından birisinin olması yeterli olur

alter table <name>
  add constraint either_email
    check (email is not null or p2pmail is not null);

Eğer sadece birisinin girildiğini garanti etmek istersek şöyle yaparız

alter table <name>
  add constraint either_email
    check (email is null <> p2pmail is not null);

Açıklaması şöyle

The second form is possibly a little confusing at first glance: what it does is compare both columns' null status -- they aren't allowed to be the same.

JSON Functions and Operators

Ortak get/extract Operators

Açıklaması şöyle

There are six get/extract operators (accessors, and serializers) which work for both json and jsonb data types (“->”, “->>”, “#>”, “#>>”). Half of these (“->”, “#>”) return json, and other half return text (“->>”, “#>>”). Note that they return NULL if the JSON doesn’t have the right structure for the request.

These operators (also range and path operators) are used in a SELECT statement ...

Sadece Jsonb Operators

Açıklaması şöyle

There are also 12 additional operators that work for jsonb only (“@>”, “<@”, “?”, “?|”, “?&”, “||”, “-”, “#-”, “@?”, “@@”).

These include:

- Containment operators (“@>” and “<@”)

- Existence operators (“?”, “?|”, “?&”)

- Concatenation operator (“||”)

- Deletion operators (“-”, “#-”)

- Path checking operators (“@?”, “@@”).

The containment and path checking operators all return boolean results, while the other operators all return jsonb results.

Processing Functions

 JSON Processing Functions yazısına taşıdım

Aggregation

Açıklaması şöyle

There are also json specific aggregation functions ( json_agg etc), which allow you to aggregate (compute a single result from multiple inputs) other data types or json/jsonb into json/jsonb. You can also use standard aggregation functions after casting json to the correct type. 

1.  -> Select Key/Value Pair

Json'daki bir alanı çekmek için kullanılır. Key denilen şey tablodaki json/jsonb sütun ismi. Value denilen şey de Json'daki alan ismidir. Value değerine text olarak değil, kendi tipinde erişiriz.

Json'daki alan isimler F1 -> F2 -> F3 şeklinde fluent olarak kullanılabilir.

Örnek

Elimizde şöyle bir tablo olsun

CREATE TABLE LunchOrders(student_id INT, orders JSONB); 

Bu tabloya veri girelim. Json verisi tek tırnak içindedir.

INSERT INTO LunchOrders VALUES(100, '{
  "order_date": "2020-12-11",
  "order_details": {
    "cost": 4.25,
    "entree": ["pizza"],
    "sides": ["apple", "fries"],
    "snacks": ["chips"]}
  }'      
);

INSERT INTO LunchOrders VALUES(100, '{
  "order_date": "2020-12-12",
  "order_details": {
    "cost": 4.89,
    "entree": ["hamburger"],
    "sides": ["apple", "salad"],
    "snacks": ["cookie"]}
  }'      
);

Örnek

Sadece bir alanı sorgulamak için şöyle yaparız.

SELECT orders -> 'order_date' FROM lunchorders WHERE student_id = 100;

Örnek

İki alanı sorgulamak için şöyle yaparız. Burada order_details altındaki sides dizisi fluent olarak sorgulanıyor

SELECT orders -> 'order_date', orders-> 'order_details' -> 'sides'
  FROM lunchorders WHERE student_id = 100;

Eğer sides dizisinin ilk elemanını istersek şöyle yaparız

SELECT orders -> 'order_date', orders-> 'order_details' -> 'sides' -> 0
  FROM lunchorders WHERE student_id = 100;

2.  ->> Filter Query Results - Nesnenin Alanına Erişir WHERE Cümlesinde Kullanılabilir

 ->> Operator - Filter Query Results yazısına taşıdım

4. #>> Filter Query Results In a Nested Object

Açıklaması şöyle

#>> operator that extracts the JSON sub-object at the specified path as text

Açıklaması şöyle

The jsonb_extract_path_text is a Postgres function that is equivalent to the  #>> operator

Şöyle yaparız

private EntityManager entityManager;

public List<Item> findAll(String expression) {
  return entityManager
    .createNativeQuery("SELECT * FROM item i WHERE " 
    + "i.jsonb_content#>>'{string_value}' LIKE '" 
    + expression + "'", Item.class)
    .getResultList();
}

Aynı şey SQL ile şöyledir

select item0_.jsonb_content as jsonb_co2_0_ 
from item item0_ 
where jsonb_extract_path_text(item0_.jsonb_content,?) like ?

5. @> Check If  an Object Contains a Value

WHERE cümlesinde Array tipler için kullanılır

Örnek

Şöyle yaparız

SELECT orders FROM lunchorders
  WHERE orders -> 'order_details' -> 'sides' @> '["salad"]'; 

orders
----------
{"order_date": "2020-12-12", "order_details": {"cost": 4.89, "sides": ["apple", "salad"],
  "entree": ["hamburger"], "snacks": ["cookie"]}}
(1 row) 

Örnek

Şöyle yaparız.

SELECT * FROM books_data WHERE data @> '{"author" : "Mark Figueroa"}';

Replication

Giriş

Primary veri tabanından diğerlerine replication işini hiç yapmadım. 

Şartlar

Açıklaması şöyle

The basic requirement replication requirement,

1. Replica DB required same PG version

2. Replica DB is in always sync with Primary DB.

Replication Lag

Örnek

Gecikmenin sebebi için bir örnek burada. Burada replica üzerinde koşan sorgu, WAL sender'i engellediği için gecikme oluyor. Özet olarak silme işlemi eskice kayıtlar üzerinde yapılsa daha iyi

You see when you have postgres replication you have a dilemma.

In one hand someone is querying standby with long running query, pinning a snapshot at t7.

On the other hand the primary has deleted some rows and vacuum is about to purge deleted tuples on t7 as no query on the primary needs them. Vacuum purges those, new WAL records of the purge.

WAL sender kicks in and send the vacuumed WAL entries to purge t7 on the standby but guess what? standby still need them. This blocks replication as the WAL receiver cannot apply the purged row if a query need them.

This creates lag in replication and as a result entire replication halt until queries are done.

Replication Lag Ölçümü

Eğer Patroni kullanmıyorsak elle ölçüm yapılabilir

Örnek

Şöyle yaparız

# Check the current LSN on primary DB.
psql -h primary_host -c "SELECT pg_current_wal_lsn();"

# check replica lag
psql -h replica_host 
  -c "SELECT pg_is_in_recovery(),pg_is_wal_replay_paused(),pg_last_wal_receive_lsn(),
    pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp();"

# Check the difference between primary DB LSN and replica replay LSN.
lsn_diff_size = psql -h replica_host 
  -c "SELECT pg_wal_lsn_diff('54/5A282990','54/5A282990');"

# Take the different and Check the size in MB or GB.
// LAG difference in MB
psql -h replica_host -c "SELECT round(lsn_diff_size/pow(1024,2.0),2 missing_lsn_mIB;" 

// LAG difference in GB
psql -h replica_host -c "SELECT round(lsn_diff_size/pow(1024,3.0),2 missing_lsn_GIB;"

Ayarlar

Not : Bir örnek burada

1. Master Ayarları

Master üzerinde bir kullanıcı yarat

CREATE USER replication 
REPLICATION LOGIN CONNECTION LIMIT 1 
ENCRYPTED PASSWORD 'replicationpa55word';

# change the maximum number of connections allowed to the replication user
ALTER ROLE replication CONNECTION LIMIT -1;

/etc/postgresql/12/main/postgresql.conf dosyasına şunu ekle. 172.XXX adresini master bilgisayarın gerçek IP adresi ile değiştir. 

listen_addresses = 'localhost,172.16.10.220'
wal_level = replica
max_wal_senders = 10
wal_keep_segments = 64

Slave bilgisayarın bağlanabilmesi için doğrulanması gerekir.  etc/postgresql/12/main/pg_hba.conf dosyasına şunu ekle. 

host    replication     replication     172.16.10.119/0   md5

Master bilgisayarı tekrar başlat

sudo /etc/init.d/postgresql restart
veya 
sudo service postgresql restart

Örnek

Şöyle yaparız

# On primary — in postgresql.conf
wal_level = replica
max_wal_senders = 5
wal_keep_size = 1GB

# On replica - recovery.conf or postgresql.conf
primary_conninfo = 'host=primary-host port=5432 user=replicator'
hot_standby = on

2. Slave Ayarları

/etc/postgresql/12/main/postgresql.conf dosyasına şunu ekle. 172.XXX adresini slave bilgisayarın gerçek IP adresi ile değiştir. 

listen_addresses = 'localhost,172.16.10.119'
wal_level = replica
max_wal_senders = 10
wal_keep_segments = 64

Master bilgisayarın bağlanabilmesi için doğrulanması gerekir.  etc/postgresql/12/main/pg_hba.conf dosyasına şunu ekle. 

host    replication     replication     172.16.10.220/0   md5

Slave bilgisayarın tüm verisini sil

sudo rm -rfv * cd /var/lib/postgresql/12/main/

Master bilgisayardaki tüm veriyi slave bilgisayara kopyala. Burada şifre isteyecektir. Şifre replicationpa55word. Bitince slave bilgisayar tekrar başlatılır

sudo su postgres \
  pg_basebackup \
  -h 172.16.10.220 \
  -U replication \
  -p 5432 \
  -D /var/lib/postgresql/12/main/  \
  -Fp \
  -Xs \
  -P \
  -R

Multi-Version Concurrency Control

Giriş

Açıklaması şöyle

Postgres was the first DBMS to rollout multi-version concurrency control (MVCC), which means reading never blocks writing and vice versa. This feature is one of the main reasons why businesses prefer Postgres to MySQL. As Postgres explains, "Unlike most other database systems which use locks for concurrency control, Postgres maintains data consistency by using a multiversion model. This means that while querying a database each transaction sees a snapshot of data (a database version) as it was some time ago, regardless of the current state of the underlying data. This protects the transaction from viewing inconsistent data that could be caused by (other) concurrent transaction updates on the same data rows, providing transaction isolation for each database session."

Utilizing what Oracle calls 'snapshot isolation', MVCC lets multiple readers and writers concurrently interact with the Postgres database, eliminating the need for a read-write lock every time someone interacts with the data. A side benefit is this process provides a big efficiency boost.

Açıklaması şöyle

When we execute a SQL statement, it uses a snapshot of data instead of every row. This prevents users from viewing inconsistent data generated by concurrent transactions. It also minimizes lock contentions for different sessions trying to read or write data.

Bir başka açıklama şöyle

Postgres works with MVCC (Multiversion Concurrency Control) that ultimately creates a new row for update/insert. This helps in concurrent transactions with older transactions having access to an older version of data(as they need to be completed) but at the same time, it’s adding a lot of stale storage in the system. So at a time, only one version of a row will be active and all others will present as dead(stale) rows.

Kısaca tüm bu açıklamalar şunu söylüyor, PostgreSQL satırların kopyasını alıyor ve bir müddet sonra bu satırlar gereksiz/geçersiz hale geliyorlar. Bu yüzden bu satırları ara ara VACUUM, REPACK Extension gibi bir şeyle temizlemek gerekiyor.

Transaction ID

Her transaction için bir tane ID üretilir. Bunu görmek için şöyle yaparız

BEGIN;
SELECT * FROM table1;
SELECT txid_current();
txid_current 
--------------
          754
(1 row)

COMMIT;

MVCC model

MVCC modelde her satır için gizli xmin ve xmax sütunları vardır. Bunları görmek için şöyle yaparız

SELECT xmin, xmax, * FROM <table_name>:
 xmin | xmax | <column_name>
------+------+---------------
  800 |    0 | <column_value>

Açıklaması şöyle

xmin holds the txid of the transaction that inserted this tuple.

xmax holds the txid of the transaction that deleted or updated this tuple. If this tuple has not been deleted or updated, t_xmax is set to 0, which means INVALID.

Örnek

Şeklen şöyle

| xmin | xmax | c1  | c_tid |
|------|------|-----|-------|
| 9    | 12   | 100 | (0,1) |
| 12   | -    | 200 | (0,2) |

Açıklaması şöyle

If transaction 9 creates a row, a new tuple is created with a header xmin value of 9. If a later transaction 12 updated the row, a new tuple is created with xmin 12 and the old tuple xmax is marked as 12 indicating that the old tuple was “alive” from 9–12. Thus the row has two tuples (row versions) one which lived between 9–12 and one that is alive starting from 12 and onwards. A repeatable read transaction 10 for instance must read the row’s old tuple 9–12 and not the new one updated by transaction 11. 

Daha sonra VACUUM çalıştırılır ve şöyle olur

| xmix | xmax | c1  | c_tid |
|------|------|-----|-------|
| 12   | -    | 200 | (0,2) |

Açıklaması şöyle

A VACUUM operation clean tuples that have been deleted AND no longer required by any running transactions.

DELETE İşlemi

Açıklaması şöyle. PostgreSQL silinmesi istenen satırları hemen silmiyor. Bu yüzden bu satırları ara ara VACUUM ile silmek gerekiyor.

When your Java application executes a DELETE or UPDATE statement against a PostgreSQL database, a deleted record is not removed immediately nor is an existing record updated in its place. Instead, the deleted record is marked as a dead tuple and will remain in storage. The updated record is, in fact, a brand new record that PostgreSQL inserts by copying the previous version of the record and updating requested columns. The previous version of that updated record is considered deleted and, as with the DELETE operation, marked as a dead tuple.

There is a good reason why the database engine keeps old versions of the deleted and updated records in its storage. For starters, your application can run a bunch of transactions against PostgreSQL in parallel. Some of those transactions do start earlier than others. But if a transaction deletes a record that still might be of interest to a few transactions started earlier, then the record needs to be kept in the database (at least until the point in time when all earlier started transactions finish). This is how PostgreSQL implements MVCC (multi-version concurrency protocol).

It’s clear that PostgreSQL can’t and doesn’t want to keep the dead tuples forever. This is why the database has its own garbage collection process called vacuuming. There are two types of VACUUM — the plain one and the full one. The plain VACUUM works in parallel with your application workloads and doesn’t block your queries. This type of vacuuming marks the space occupied by dead tuples as free, making it available for new data that your app will add to the same table later. The plain VACUUM doesn’t return the space to the operating system so that it can be reused by other tables or 3rd party applications (except in some corner cases when a page includes only dead tuples and the page is in the end of a table).

Sütun Tipleri - json

Giriş

Json verisi saklamak için kullanılır. Açıklaması şöyle.

The only difference between json & jsonb is their storage:

- json is stored in its plain text format, while
- jsonb is stored in some binary representation

There are 3 major consequences of this:

- jsonb usually takes more disk space to store than json (sometimes not)
- jsonb takes more time to build from its input representation than json
- json operations take significantly more time than jsonb (& parsing also needs to be done each time you do some operation at a json typed value)

When jsonb will be available with a stable release, there will be two major use cases, when you can easily select between them:

- If you only work with the JSON representation in your application, PostgreSQL is only used to store & retrieve this representation, you should use json.
- If you do a lot of operations on the JSON value in PostgreSQL, or use indexing on some JSON field, you should use jsonb.

Açıklaması şöyle. json tipi 2012 yılında PostgreSQL 9.2 ile eklendi. Yani jsonb sütün tipinden daha önce eklendi. 

In 2012, PostgreSQL 9.2 introduced the first JSON data type in Postgres. It had syntax validation but it stored the incoming document directly as text with white spaces included. It wasn't very useful for real-world querying, index-based searching, and other functionalities you would normally do with a JSON document.

Açıklaması şöyle.

...a JSON datatype (2012), and a potpourri of new features in PostgreSQL 10 (better native support for partitioning and replication, full text search support for JSON, etc.)

Örnek

Şöyle yaparız

CREATE TABLE IF NOT EXISTS posts (
   ...
   metadata json default '{}'
 );