Tuesday, 17 June 2025

MS SQL Server JSON Basics Cheat Sheet

 MS SQL Server JSON Basics Cheat Sheet

This cheat sheet provides a quick reference for common functions and syntax used to work with JSON data in Microsoft SQL Server (SQL Server 2016 and later).

1. Validating JSON Text

Use ISJSON() to check if a string contains valid JSON. It returns 1 for valid JSON, 0 for invalid JSON, and NULL if the input is NULL.

-- Example Data
DECLARE @json_valid NVARCHAR(MAX) = N'{"name": "John Doe", "age": 30}';
DECLARE @json_invalid NVARCHAR(MAX) = N'{"name": "Jane Doe", "age": }'; -- Invalid JSON

-- Check Validity
SELECT ISJSON(@json_valid) AS IsValid1;         -- Returns 1
SELECT ISJSON(@json_invalid) AS IsValid2;       -- Returns 0
SELECT ISJSON(NULL) AS IsNullValid;         -- Returns NULL

2. Extracting Scalar Values

Use JSON_VALUE() to extract a single scalar value (string, number, boolean) from a JSON string using a JSON path expression.

       Path Syntax: $ represents the root object. Use . for member access and [index] for array element access (0-based).

-- Example Data
DECLARE @json_data NVARCHAR(MAX) = N'{
  "person": {
    "name": "Alice",
    "age": 28,
    "isStudent": false,
    "address": {
      "street": "123 Main St",
      "city": "Anytown"
    },
    "courses": ["Math", "History"]
  }
}';

-- Extract Scalar Values
SELECT
    JSON_VALUE(@json_data, '$.person.name') AS PersonName,       -- Returns 'Alice'
    JSON_VALUE(@json_data, '$.person.age') AS PersonAge,         -- Returns '28' (as string by default)
    JSON_VALUE(@json_data, '$.person.isStudent') AS IsStudent,   -- Returns 'false'
    JSON_VALUE(@json_data, '$.person.address.city') AS City,     -- Returns 'Anytown'
    JSON_VALUE(@json_data, '$.person.courses[0]') AS FirstCourse; -- Returns 'Math'

-- Extract with specific data type (using CAST/CONVERT)
SELECT
    CAST(JSON_VALUE(@json_data, '$.person.age') AS INT) AS PersonAgeInt,
    CAST(JSON_VALUE(@json_data, '$.person.isStudent') AS BIT) AS IsStudentBit;

3. Extracting Objects or Arrays

Use JSON_QUERY() to extract an object or an array from a JSON string. JSON_VALUE returns NULL if it targets an object or array.

-- Example Data (using @json_data from above)

-- Extract Object
SELECT JSON_QUERY(@json_data, '$.person.address') AS AddressObject;
-- Returns: {"street": "123 Main St", "city": "Anytown"}

-- Extract Array
SELECT JSON_QUERY(@json_data, '$.person.courses') AS CoursesArray;
-- Returns: ["Math", "History"]

-- Extracting an object/array with JSON_VALUE returns NULL
SELECT JSON_VALUE(@json_data, '$.person.address') AS AddressValueIsNull; -- Returns NULL

4. Modifying JSON Data

Use JSON_MODIFY() to update a property value, insert a new property, or delete a property within a JSON string. It returns the modified JSON string.

       Path Syntax: Same as JSON_VALUE and JSON_QUERY.

       append keyword: Adds an element to the end of an array.

       lax mode (default): If the path doesn't exist for an update/delete, it doesn't raise an error.

       strict mode: If the path doesn't exist for an update/delete, it raises an error.

-- Example Data
DECLARE @json_doc NVARCHAR(MAX) = N'{"name": "Bob", "age": 35, "skills": ["SQL"]}';

-- Update existing value
SET @json_doc = JSON_MODIFY(@json_doc, '$.age', 36);
-- @json_doc is now: {"name": "Bob", "age": 36, "skills": ["SQL"]}

-- Insert new key/value pair
SET @json_doc = JSON_MODIFY(@json_doc, '$.city', 'New York');
-- @json_doc is now: {"name": "Bob", "age": 36, "skills": ["SQL"], "city": "New York"}

-- Append to an array
SET @json_doc = JSON_MODIFY(@json_doc, 'append $.skills', 'JSON');
-- @json_doc is now: {"name": "Bob", "age": 36, "skills": ["SQL", "JSON"], "city": "New York"}

-- Delete a key/value pair (set value to NULL)
SET @json_doc = JSON_MODIFY(@json_doc, '$.city', NULL);
-- @json_doc is now: {"name": "Bob", "age": 36, "skills": ["SQL", "JSON"]}

SELECT @json_doc AS ModifiedJson;

5. Parsing JSON into Rows and Columns

Use OPENJSON() to parse JSON text and return objects and properties as rows and columns. It's often used with a WITH clause to define the output schema.

       Default Schema: Returns key/value pairs and type information.

       Explicit Schema (WITH clause): Defines columns and their data types, mapping them to JSON properties using JSON paths.

-- Example Data
DECLARE @json_array NVARCHAR(MAX) = N'[
  {"id": 1, "name": "Product A", "price": 10.50},
  {"id": 2, "name": "Product B", "price": 25.00, "tags": ["tech", "gadget"]},
  {"id": 3, "name": "Product C", "price": 5.75}
]';

-- Using OPENJSON with Default Schema
SELECT [key], [value], [type]
FROM OPENJSON(@json_array);
/* Returns rows representing each object in the array:
key | value                                                     | type
----|-----------------------------------------------------------|------
0   | {"id": 1, "name": "Product A", "price": 10.50}            | 5 (Object)
1   | {"id": 2, "name": "Product B", "price": 25.00, "tags":...}| 5 (Object)
2   | {"id": 3, "name": "Product C", "price": 5.75}            | 5 (Object)
*/

-- Using OPENJSON with Explicit Schema (WITH clause)
SELECT *
FROM OPENJSON(@json_array)
WITH (
    ProductID INT '$.id',
    ProductName VARCHAR(100) '$.name',
    Price DECIMAL(10, 2) '$.price',
    Tags NVARCHAR(MAX) '$.tags' AS JSON -- Keep nested array as JSON
);
/* Returns:
ProductID | ProductName | Price | Tags
----------|-------------|-------|---------------------
1         | Product A   | 10.50 | NULL
2         | Product B   | 25.00 | ["tech", "gadget"]
3         | Product C   | 5.75  | NULL
*/

-- Accessing nested arrays within OPENJSON
DECLARE @json_single_obj NVARCHAR(MAX) = N'{
  "orderId": 101,
  "customer": "Jane Smith",
  "items": [
    {"sku": "X1", "qty": 2},
    {"sku": "Y2", "qty": 1}
  ]
}';

SELECT i.*
FROM OPENJSON(@json_single_obj, '$.items') -- Specify the path to the array
WITH (
    SKU VARCHAR(10) '$.sku',
    Quantity INT '$.qty'
) AS i;
/* Returns:
SKU | Quantity
----|----------
X1  | 2
Y2  | 1
*/

6. Formatting T-SQL Results as JSON

Use FOR JSON to format query results as JSON text.

       FOR JSON PATH: Gives you full control over the output format using dot notation in column aliases. Nested structures are created based on these aliases.

       FOR JSON AUTO: Automatically creates nested JSON structures based on the table structure in the SELECT statement and the order of columns.

-- Example Table Data (Assume a table named 'Employees')
-- EmployeeID | FirstName | LastName | Department | ManagerID
-- -----------|-----------|----------|------------|-----------
-- 1          | John      | Doe      | Sales      | NULL
-- 2          | Jane      | Smith    | HR         | 1
-- 3          | Peter     | Jones    | Sales      | 1

-- Using FOR JSON PATH
SELECT
    EmployeeID AS 'employee.id',
    FirstName AS 'employee.firstName',
    LastName AS 'employee.lastName',
    Department AS 'employee.dept'
FROM Employees
WHERE EmployeeID = 1
FOR JSON PATH, ROOT('employees'); -- ROOT adds a root element
-- Returns: {"employees":[{"employee":{"id":1,"firstName":"John","lastName":"Doe","dept":"Sales"}}]}

-- Using FOR JSON AUTO
SELECT
    EmployeeID,
    FirstName,
    LastName,
    Department
FROM Employees
WHERE Department = 'Sales'
FOR JSON AUTO;
-- Returns: [{"EmployeeID":1,"FirstName":"John","LastName":"Doe","Department":"Sales"},{"EmployeeID":3,"FirstName":"Peter","LastName":"Jones","Department":"Sales"}]

-- FOR JSON PATH with Nested Structure
SELECT
    e.EmployeeID AS 'id',
    e.FirstName AS 'name.first',
    e.LastName AS 'name.last',
    (SELECT m.FirstName + ' ' + m.LastName
     FROM Employees m
     WHERE m.EmployeeID = e.ManagerID) AS 'managerName'
FROM Employees e
WHERE e.EmployeeID = 2
FOR JSON PATH, WITHOUT_ARRAY_WRAPPER; -- WITHOUT_ARRAY_WRAPPER removes the outer [] for single results
-- Returns: {"id":2,"name":{"first":"Jane","last":"Smith"},"managerName":"John Doe"}

This cheat sheet covers the fundamental JSON functions in SQL Server. Remember to consult the official Microsoft documentation for more advanced features and options.

Monday, 7 August 2023

Performance Tuning SQL Server Queries: Boosting Efficiency and Speed

Introduction:

In the realm of SQL Server database administration, performance tuning plays a pivotal role in enhancing the efficiency and speed of queries. As the volume of data grows and application demands increase, optimizing the execution of queries becomes critical to maintain the overall system performance. This article delves into the fundamental aspects of performance tuning for SQL Server queries, exploring techniques to identify bottlenecks and implement optimizations for optimal query performance.


1. Understanding Query Execution Plan:

The query execution plan serves as a roadmap to understanding how SQL Server processes a query. By analyzing the execution plan, administrators can identify potential performance issues and determine the most efficient way to execute a query. We will discuss the use of SQL Server Management Studio (SSMS) and dynamic management views (DMVs) to obtain query execution plans and analyze them.


2. Indexing Strategies:

Appropriate indexing is one of the most powerful tools for optimizing query performance. We'll explore various indexing strategies, including clustered, non-clustered, and covering indexes, as well as their impact on query execution. Additionally, we'll delve into common indexing pitfalls and best practices to ensure the right indexes are created for your workload.


3. Query Optimization Techniques:

In this section, we'll explore a range of query optimization techniques, such as JOIN optimization, subquery optimization, and using appropriate SQL Server functions efficiently. We'll also discuss how to leverage query hints and plan guides to influence the execution plan when needed, though caution must be exercised when using these features.


4. Query Rewriting and Refactoring:

Sometimes, rewriting or refactoring a query can lead to significant performance gains. We'll discuss common SQL coding patterns that can be improved and explore how to rewrite complex queries into simpler and more efficient forms. Additionally, we'll cover the importance of avoiding correlated subqueries and employing CTEs (Common Table Expressions) for better query performance.


5. Temporal and Cached Data:

Caching query results and employing temporal tables can improve performance significantly by reducing the load on the database server. We'll delve into techniques for caching data, both at the application level and within SQL Server itself, to reduce query execution times and improve overall user experience.


6. Monitoring and Profiling:

To ensure ongoing performance optimization, monitoring and profiling queries are essential. We'll explore SQL Server's built-in tools, such as SQL Server Profiler and Extended Events, to capture query performance data. Additionally, we'll discuss how to leverage performance monitoring tools like SQL Server Performance Monitor to analyze resource utilization and identify performance bottlenecks.


Conclusion:

Performance tuning of SQL Server queries is a continuous process that requires a comprehensive understanding of the database structure, indexing strategies, and query execution plans. By implementing the techniques discussed in this article, you can significantly improve the efficiency and speed of your queries, resulting in better application performance and a more positive user experience. Remember to analyze the impact of any changes thoroughly and continuously monitor your system to ensure long-term success in performance optimization.

Sunday, 23 May 2021

SQL Server with Python - An Intro

Everyone uses SQL, and everyone uses Python. SQL is the de-facto standard for databases. Python on the other hand is an all-star, a top language for data analytics, machine learning, and web development. Imagine both, together.

This is actually incredibly easy to setup. We can quickly utilize the dynamic nature of Python, to control and build queries in SQL. The best part? After set-up, you don’t need to do anything.

Both of these amazing tools together, allow us to reach new heights of automation and efficiency.

pyodbc

Our bridge between the two technologies is pyodbc. This library allows easy access to ODBC databases.

ODBC, short for Open Database Connectivity, is a standardised application programming interface (API) for accessing databases, developed by the SQL Access group back in the early 90's.

Compliant database management systems (DBMS) include:

  • IBM Db2
  • MS Access
  • MS SQL Server
  • MySQL
  • Oracle

In this article, we will be using MS SQL Server. For the most part, this should be directly transferable for use with any ODBC compliant database. The only change required should be with the connection setup.

Connect

The first thing we need to do is create a connection to the SQL server. We can do this using pyodbc.connect. Within this function we must also pass a connection string.

This connection string must specify the DBMS Driver, the Server, a specific Database to connect to, and our connection settings.

So, lets assume we want to connect to server UKXXX00123,45600, database DB01 , to do this we want to use SQL Server Native Client 11.0.

We will be connecting from an internal, and thus trusted connection (we do not need to enter our username and password).

cnxn_str = ("Driver={SQL Server Native Client 11.0};"
"Server=UKXXX00123,45600;"
"Database=DB01;"
"Trusted_Connection=yes;")

Our connection is now initialized with:

cnxn = pyodbc.connect(cnxn_str)

If we are not accessing the database via a trusted connection, we will need to enter the username and password that we would usually use to access the server via SQL Server Management Studio (SSMS).

For example, if our username is JoeBloggs, and our password is Password123, we should immediately change our password.

But before changing that horrible password, we can connect like so:

cnxn_str = ("Driver={SQL Server Native Client 11.0};"
"Server=UKXXX00123,45600;"
"Database=DB01;"
"UID=JoeBloggs;"
"PWD=Password123;")
cnxn = pyodbc.connect(cnxn_str)

Now we are connected to the database, we can begin performing SQL queries via Python.

Run a Query

Every query we run on SQL Server now will consist of a cursor initialization, and query execution. Additionally, if we make any changes inside the server, we also need to commit these changes to the server (which we cover in the next section).

To initialize a cursor:

cursor = cnxn.cursor()

Now, whenever we want to perform a query, we use this cursor object.

Let’s first select the top 1000 rows from a table called customers:

cursor.execute("SELECT TOP(1000) * FROM customers")

This performs the operation, but within the server, and so nothing is actually returned to Python. So let’s look at extracting this data from SQL.

Extract Data

To extract our data from SQL into Python, we use pandas. Pandas provides us with a very convenient function called read_sql, this function, as you may have guessed, reads data from SQL.

read_sql requires both a query and the connection instance cnxn, like so:

data = pd.read_sql("SELECT TOP(1000) * FROM customers", cnxn)

This returns a dataframe containing the top 1000 rows from the customers table.

Changing Data in SQL

Now, if we wanted to change the data in SQL, we need to add another step to the original initialize connection, execute query process.

When we execute queries in SQL, these changes are kept in a temporarily existing space, they are not made directly to the data.

To make these changes permanent, we must commit them. Lets concatenate the firstName and lastName columns, to create a fullName column.

cursor = cnxn.cursor()# first alter the table, adding a column
cursor.execute("ALTER TABLE customer " +
"ADD fullName VARCHAR(20)")
# now update that column to contain firstName + lastName
cursor.execute("UPDATE customer " +
"SET fullName = firstName + " " + lastName")

At this point, fullName does not exist in our database. We must commit these changes to make them permanent:

cnxn.commit()

Saturday, 22 May 2021

Docker – Installation on Windows

A quick walk through of Docker installation on Windows 10.

1. Downloading the binaries


Download the binaries from the official website.

2. Install WSL 2

Install WSL 2 following instructions provided in the Microsoft website. This step can be done before installing the docker or alternatively we can let docker installer install it for us. Windows Sub System for Linux provides environment to run GNU/Linux environment , command line tools , utilities directly on Windows without installing a VM or dual boot .

3. Installing Docker

Go to the place where we have downloaded the Docker Desktop Installer.exe , right click Run as administrator.
Unlike other installers , Docker installer will not prompt us with many options . The only two options that we will see during installation are “Install required Windows components for WSL 2 “ and “Add shortcut to desktop”. Select both the options and click ok to start the installation.

Wait for the installation to go through.

Once the installation is successful , we will see the following screen :

4. Starting Docker Desktop

Before we can execute any Docker command, we have to first start the Docker desktop. We can start the Docker Desktop from the start menu or from the desktop icon.

The status of the Docker engine is shown at the bottom left corner, it should be green before we can interact with Docker cli .

5. Optional Configuration


There are some configurations in Docker , which are good to have right from the start . Click on the gear icon on the top right side of the Docker Desktop dashboard .

Enabling Kubernetes (Optional)

Navigate to Kubernetes tab and then click on the Enable Kubernetes check box and then click on “Apply & Restart”. This is the fastest way to bring up k8s on our machine.

Enabling WSL 2 Integration

If we have installed any other operating system using WSL2, we don’t have to install Docker once again. We can use the same installation here also by enabling the integration.
Go to settings > Resources > WSL Integration > Enable the integration for the installed OS .

6. Verification

Now that installation is complete, we verify the installation using the following command :
docker info
This will display the details of of Docker Desktop .


For more on this , visit official documentation .