One of my existing clients has sent me an interesting question. In this blog, I would share one interesting situation where my client has created an availability group and it was visible in SSMS, but no role was visible in failover cluster manager.

Here is the picture which you explain what we were seeing.

Generally, when an availability group is created, an AG resource is created in a Windows cluster.

WORKAROUND/SOLUTION

The question which was asked by my client was perfectly valid. It is possible to run into the situation if we do below steps.

1. Enable feature of AlwaysOn availability group without creating windows cluster.
   
2. Create Availability Group and choose NONE as cluster type as there is no option called “NONE” as cluster type in below screen.
   
3. Now, enabled windows clustering feature and create windows cluster.

At this point, you can open cluster manager and would see no availability group role because availability group was created as Cluster Type None.

So, if you ever see such situation, check the Dashboard of Always On availability group and have a look at cluster type which is highlighted as below.

This feature is called as a Read Scale and you can read more about this at below link Use read-scale with Always On availability groups

Hope you found this information useful. Please comment and let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Availability Group Seen in SSMS but missing in Cluster Manager. What’s Wrong?

Sometimes if the sequence is not followed correctly, we might see some weird errors. In this blog, we would learn how to fix always on an error while adding new replica “The specified instance of SQL Server is hosted by a system that is not a Windows Server Failover Cluster(WSFC) node.”

This was one of my existing clients to whom I worked and configured Always On Availability Group. They had some issues with nodes, and they ended up in rebuilding the cluster.

Later, they were able to redeploy and bring the cluster back. Now, when they were trying to add a replica Node2 from Node1 via SQL Server Management Studio, an error was displayed.

Here is the text of the error message

The specified instance of SQL Server is hosted by a system that is not a Windows Server Failover Cluster(WSFC) node. (Microsoft.SqlServer.Management.HadrTasks)
 ——————————
Program Location:
 at Microsoft.SqlServer.Management.Hadr.SelectReplicasController.VerifyConnection(UIConnectionInfo ci, IServerType serverType)
   at Microsoft.SqlServer.Management.UI.ConnectionDlg.Connector.ConnectionThreadUser()

From the message is clear that there is something wrong between SQL and Cluster communication. I asked them to run below command on all nodes which are part of the windows cluster.

SELECT *
FROM sys.dm_hadr_cluster_members
GO
SELECT *
FROM sys.dm_hadr_cluster

When we ran the query on the node which was giving an error, there were no rows.

WORKAROUND/SOLUTION

We disabled Always On Availability Group on the “bad” node using SQL Server Configuration Manager. Once done, we restarted the SQL Service. Then we enabled it again followed by another SQL Service restart. After finishing this, we were able to get information about the cluster and nodes via the query which I mentioned earlier.

Have you seen such an issue earlier? What were the steps to break it?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: The specified instance of SQL Server is hosted by a system that is not a Windows Server Failover Cluster(WSFC) Node.

I have helped many clients in deploying Always On Availability Group. Based on their requirement they keep using additional features provided by availability groups. This time they wanted to use always-on availability group read-only routing feature. Their goal was to offload read-only workload to the secondary replica. This was failing with error: Client unable to establish a connection because an error was encountered during handshakes before login.

I informed them that while making a connection from the application, they need to make sure of three things.

1. The routing URL is setup correctly in SQL Server.
2. The routing list is setup correctly in SQL Server.
3. Connect to the listener in the connection string.
4. Provide default database name in the connection string.
5. Provide application intent parameter in the connection string.

As per them, the above things were checked already so they wanted me to look into it and fix it.

Here is the error message which they were getting while connecting to the listener using read-only intent.

Client unable to establish connection because an error was encountered during handshakes before login. Common causes include client attempting to connect to an unsupported version of SQL Server, server too busy to accept new connections or a resource limitation (memory or maximum allowed connections) on the server..
Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server: TCP Provider: An existing connection was forcibly closed by the remote host.
Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server: Client unable to establish connection.
Sqlcmd: Error: Microsoft ODBC Driver 13 for SQL Server: Client unable to establish connection due to prelogin failure.

Here is the image

WORKAROUND/SOLUTION

Without wasting a lot of time, I asked them to show me the routing via SSMS.

This UI the Availability Group properties window. If you are using a lower version of SSMS then you might want to use the catalog views to query them. The latest SSMS can be downloaded free from below link

Download SQL Server Management Studio (SSMS).

As we can see, the routing URL is setup incorrectly. The port in the routing URL should be the port on which SQL connections are made to the instance. Typically, for default instance, it is 1433.

SQL SERVER – Find Port SQL Server is Listening – Port SQL Server is Running

As soon as the URL was changed, the read-only routing started working like a charm.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Read Only Routing Error: Client Unable to Establish Connection Because an Error was Encountered During Handshakes Before Login

While working with one of the clients to recover from a disaster, I encountered an error. In this blog we would talk about error Drop failed for Availability Group. If you want any such assistance, you can hire me for quick consultation using On Demand offering.

THE SITUATION

When I got a call from my client, they were into the situation where availability group was not coming online. After an investigation, we concluded that their windows cluster was having issues and it was unable to start. The solution in such disaster is to start windows cluster in force quorum mode.

Once we started to cluster it in ForceQuorum mode (also called as fix quorum) we tried deleting the AG and received below error.

Failed to destroy the Windows Server Failover Clustering group corresponding to availability group ‘AG’.  The operation encountered SQL Server error 41000 and has been terminated.  Refer to the SQL Server error log for details about this SQL Server error and corrective actions.
An error occurred while removing availability group ‘AG’. The DROP AVAILABILITY GROUP command removed the availability group configuration from the local metadata. However, the attempt to remove this configuration from the Windows Server Failover Clustering (WSFC) cluster failed because the Always On Availability Groups manager is not online (SQL Server error: 41081). To remove the availability group configuration from the WSFC cluster, re-enter the command. (Microsoft SQL Server, Error: 41081)

In above error message, we see another error number: 41000 for which the text is as follows. “Failed to obtain the local Windows Server Failover Clustering (WSFC) handle (Error code %d).  If this is a WSFC availability group, the WSFC service may not be running or may not be accessible in its current state.  Otherwise, contact your primary support provider.  For information about this error code, see “System Error Codes” in the Windows Development documentation.”

WHAT YOU SHOULD DO?

Well, no action needed. The message raised is not a dangerous error message. It clearly mentions that availability group information has been removed from SQL Server, but it might still exist in the failover cluster. So, you can click OK on message and check availability group in SSMS and it should be gone from “Always On High Availability” > “Availability Group”. At this point, the database would be in restoring state and you need to bring them online using below

ALTER DATABASE <DBName> SET ONLINE

OR

RESTORE DATABASE <DBName> WITH RECOVERY

Once things are back to normal, you need to reconfigure availability again.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Drop failed for Availability Group – Failed to Destroy the Windows Server Failover Clustering Group Corresponding to Availability Group

I love blog post which just solves a simple point and immediately gets us the necessary answer. Recently during one of the consulting engagement Comprehensive Database Performance Health Check at one of my customer place we identified a wait statistics which was related to SQL Server Always On Availability Group. The wait statistics issue was VDI_CLIENT_OTHER and the way to solve is blogged here SQL SERVER – Many BACKGROUND Threads with Command VDI_CLIENT_WORKER. We needed to remove automatic seeding to reduce the said wait statistics.

SQL Server 2016 introduced automatic seeding of availability groups. Here is the script which we used to list all the databases which are part of the AlwaysOn Availability Group and it’s a type of sedding.

SELECT 
	dhas.start_time
	,dhas.completion_time
	,ag.name
	,adb.database_name
	,dhas.current_state
	,dhas.performed_seeding
	,dhas.failure_state
	,dhas.failure_state_desc
FROM sys.dm_hadr_automatic_seeding as dhas 
JOIN sys.availability_databases_cluster as adb 
    ON dhas.ag_db_id = adb.group_database_id
JOIN sys.availability_groups as ag 
    ON dhas.ag_id = ag.group_id

Additionally, here is the script which you can use to monitor automatic seeding.

SELECT start_time, completion_time is_source, current_state,
failure_state, failure_state_desc
FROM sys.dm_hadr_automatic_seeding

Remember that while your AlwaysOn Availability Group database is configured for automatic seeding your SQL Server performs a backup over the network for initialization. If your database is large or your network has huge latency, auto seeding may face significant slowness and during those scenarios, you may want to disable it.

Let me know if you have ever faced performance slowness when you have enabled auto seeding. In most of the cases, I have found it to be very much helpful to your AlwaysOn Availability Scenario.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – T-SQL Script to List Automatic Seeding of Always On Availability Group

While deploying Always On Availability Group with a client, they found that when they use automatic seeding, the database was not shown on secondary. After digging more, we found that it was there earlier in “Restoring” state when seeding was in progress but then it automatically disappeared from the list of databases on secondary.  Here is the option in the wizard.

Here are a few earlier blogs which I wrote about seeding failures.

SQL SERVER – AlwaysOn Automatic Seeding – Database Stuck in Restoring State

SQL SERVER – AlwaysOn Automatic Seeding Failure – Failure_code 15 and Failure_message: VDI Client Failed

I checked SQL Server ERRORLOG to figure out what might have gone wrong with auto-seeding.

SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

The important messages about the database, which we tried to seed, are as follows.

Error: 1412, Severity: 16, State: 211.
The remote copy of database “DB” has not been rolled forward to a point in time that is encompassed in the local copy of the database log.
Starting up database ‘DB’.
The database ‘DB’ is marked RESTORING and is in a state that does not allow recovery to be run.
Automatic seeding of availability database ‘DB’ in availability group ‘AG’ failed with an unrecoverable error. Correct the problem, then issue an ALTER AVAILABILITY GROUP command to set SEEDING_MODE = AUTOMATIC on the replica to restart seeding.

The last bullet has an interesting message and tells who to restart the seeding. The very first message (Error: 1412, Severity: 16, State: 211.) tells us the cause of the problem.

You need to monitor seeding using DMVs. There are two dynamic management views (DMVs) for monitoring seeding: sys.dm_hadr_automatic_seeding and sys.dm_hadr_physical_seeding_stats. One of them shows information about seedings happening currently and “stats” show the historical data about seeding. In this case, seeding failed with SQL Error.

WORKAROUND/SOLUTION

The cause of the problem was an additional log backup which happened while seeding was in progress.  I used my own earlier blog to find backup history SQL SERVER – Get Database Backup History for a Single Database

After running the script from my blog (on ALL replicas), we found that a backup job which was taking log backup on “another” replica every 15 minutes. We disabled the job and started seeding again. This time it worked without any issue.

Have you encountered some error during Always On Availability Group? Please share via comments and help others.

Reference: Pinal Dave (https://blog.SQLAuthority.com)

First appeared on SQL SERVER – Automatic Seeding of Availability Database ‘DB’ in Availability Group ‘AG’ Failed With an Unrecoverable Error

While preparing for a disaster recovery demo for my client, I observed an interesting behavior. In this blog, I would show a quick demo where you can learn how the full backup size can be different on the primary and secondary replica for an availability database in Always On availability group.

Here is the quick script to understand the behavior.

1. Create a Database and take a full backup and create a wide table (8KB char column)

   USE master
   GO
   CREATE DATABASE SQLAGDB
   GO
   BACKUP DATABASE SQLAGDB TO DISK = 'SQLAGDB.bak'
   GO
   USE SQLAGDB
   GO
   CREATE TABLE SQLAuthority (j CHAR(8000))
   GO
   

2. Configure AlwaysOn Availability Group using SSMS – Always On Wizard for database SQLAGDB. Makes sure it is healthy and secondary in sync.
3. Insert lots of rows in the table. You can use the below script

   -- connect to database 
   USE SQLAGDB
   GO
   -- insert 100 rows
   SET NOCOUNT ON
   GO
   INSERT INTO SQLAuthority
   VALUES ('A') 
   GO 100
   -- 
   SET NOCOUNT ON
   GO
   -- insert from same table to same table (8 times)
   -- total rows should be 100+100+200+400+800+1600+3200+6400+12800
   INSERT SQLAuthority
   SELECT *
   FROM SQLAuthority 
   GO 8
   SELECT COUNT(*) from SQLAuthority -- 25600
   

4. Take COPY_ONLY backup from both replicas.

   --primary 
   :connect sqlserver-0
   BACKUP DATABASE SQLAGDB TO DISK = '\\sqlserver-0\share\primary_before_log_backup.bak'
   WITH copy_only,format
   GO
   -- secondary 
   :connect sqlserver-1
   BACKUP DATABASE SQLAGDB TO DISK = '\\sqlserver-0\share\secondary_before_log_backup.bak'
   WITH copy_only,format
   GO
   

   Here is the output
   Connecting to sqlserver-0…
   Processed 25944 pages for database ‘SQLAGDB’, file ‘SQLAGDB’ on file 1.
   Processed 28629 pages for database ‘SQLAGDB’, file ‘SQLAGDB_log’ on file 1.
   BACKUP DATABASE successfully processed 54573 pages in 6.755 seconds (63.115 MB/sec).
   Disconnecting connection from sqlserver-0…
   Connecting to sqlserver-1…
   Processed 25944 pages for database ‘SQLAGDB’, file ‘SQLAGDB’ on file 1.
   Processed 2 pages for database ‘SQLAGDB’, file ‘SQLAGDB_log’ on file 1.
   BACKUP DATABASE successfully processed 25946 pages in 2.259 seconds (89.729 MB/sec).
   Disconnecting connection from sqlserver-1…
   Notice processed pages for ‘SQLAGDB_log’.

5. Let’s take a log backup (on any replica)

   BACKUP LOG [SQLAGDB] TO  
   DISK = N'\\sqlserver-0\share\Log-backup.trn' 
   WITH NOFORMAT, NOINIT,  SKIP, NOREWIND, NOUNLOAD,  STATS = 10
   GO
   

   Processed 28630 pages for database ‘SQLAGDB’, file ‘SQLAGDB_log’ on file 1.
   BACKUP LOG successfully processed 28630 pages in 3.533 seconds (63.309 MB/sec).

6. Now take the same full backup on both replicas.

   --primary 
   :connect sqlserver-0
   BACKUP DATABASE SQLAGDB TO DISK = '\\sqlserver-0\share\primary_after_log_backup.bak'
   WITH copy_only,format
   GO
   -- secondary 
   :connect sqlserver-1
   BACKUP DATABASE SQLAGDB TO DISK = '\\sqlserver-0\share\secondary_after_log_backup.bak'
   WITH copy_only,format
   GO
   

   and here is the output:
   Connecting to sqlserver-0…
   Processed 25944 pages for database ‘SQLAGDB’, file ‘SQLAGDB’ on file 1.
   Processed 4 pages for database ‘SQLAGDB’, file ‘SQLAGDB_log’ on file 1.
   BACKUP DATABASE successfully processed 25948 pages in 3.182 seconds (63.706 MB/sec).
   Disconnecting connection from sqlserver-0…
   Connecting to sqlserver-1…
   Processed 25944 pages for database ‘SQLAGDB’, file ‘SQLAGDB’ on file 1.
   Processed 2 pages for database ‘SQLAGDB’, file ‘SQLAGDB_log’ on file 1.
   BACKUP DATABASE successfully processed 25946 pages in 2.365 seconds (85.707 MB/sec).
   Disconnecting connection from sqlserver-1…
   

7. A picture is worth a thousand words:
   

This situation would come when we don’t perform log backup and T-Log continues to grow. From above we can understand that full backup with copy_only has different logic to take a portion of the log file on primary and secondary. Interesting enough the additional size (208 + 229 = 437) is equal to LDF size of the database file. Once log backup is taken there is no additional log file to be backed up and size became the same. I hope you learn about Different Size of COPY_ONLY Full Backup on Primary and Secondary Replica in Always On.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Different Size of COPY_ONLY Full Backup on Primary and Secondary Replica in Always On

I always recommend my client to go with the latest and great version of tools like SQL Server Management Studio (SSMS). In this blog, we would learn about a situation where an older version of SSMS was giving an error while adding a new database to existing availability group – This BACKUP or RESTORE command is not supported on a database mirror or secondary replica.

THE SITUATION

My client had an existing availability group which was set up and deployed by a vendor. Now, they wanted to add a new database to an existing availability group. They were using the wizard to add the database. They selected the option to do a fresh backup/restore. Due to old SSMS version, automatic seeding option was not available. Here was the error which they were getting in the UI.

Here is the text of error message:

Restoring database log resulted in an error. (Microsoft.SqlServer.Management.HadrTasks)
——————————
ADDITIONAL INFORMATION:
Restore failed for Server ‘SQLSERVER-1’. (Microsoft.SqlServer.SmoExtended)
——————————
System.Data.SqlClient.SqlError: This BACKUP or RESTORE command is not supported on a database mirror or secondary replica. (Microsoft.SqlServer.Smo)

The surprising part was that even after the error came, the database was added to the availability group successfully. This means the error was a benign error and can be ignored but I want to know what error was appearing. To dig further I generated the script of Always On and executed it manually. It failed with the exact same error.

WORKAROUND/SOLUTION for RESTORE command

It was clear that the database was getting added successfully due to automatic seeding. We can see below in ERRORLOG.

Primary Replica:

• Database backed up. Database: OneMoreDatabase, creation date(time): 2019/06/11(13:21:51), pages dumped: 355, first LSN: 38:2800:1, last LSN: 38:2824:1, number of dump devices: 1, device information: (FILE=1, TYPE=DISK: {‘\\sqlserver-0\Share\OneMoreDatabase.bak’}). This is an informational message only. No user action is required.
• Database backed up. Database: OneMoreDatabase, creation date(time): 2019/06/11(13:21:51), pages dumped: 522, first LSN: 38:2712:1, last LSN: 38:2824:1, number of dump devices: 1, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {‘{3AFC5950-4E79-4225-B572-0A70AD876E45}’}). This is an informational message only. No user action is required.
• Log was backed up. Database: OneMoreDatabase, creation date(time): 2019/06/11(13:21:51), first LSN: 38:2712:1, last LSN: 38:2824:1, number of dump devices: 1, device information: (FILE=1, TYPE=DISK: {‘\\sqlserver-0\Share\OneMoreDatabase_20190626054651.trn’}). This is an informational message only. No user action is required.

Secondary Replica:

• Database was restored: Database: OneMoreDatabase, creation date(time): 2019/06/11(13:21:51), first LSN: 38:2712:1, last LSN: 38:2824:1, number of dump devices: 1, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {‘{E9D35CEB-D6FD-40D6-A73C-06560EFD95D9}’}). Informational message. No user action required.
• Database was restored: Database: OneMoreDatabase, creation date(time): 2019/06/11(13:21:51), first LSN: 38:2800:1, last LSN: 38:2824:1, number of dump devices: 1, device information: (FILE=1, TYPE=DISK: {‘\\sqlserver-0\Share\OneMoreDatabase.bak’}). Informational message. No user action required.

If we look at the backup and restore commands, there are two backups. Once on TYPE=DISK and another on TYPE=VIRTUAL_DEVICE. The second backup comes when there is an automatic seeding happening for the database.

Running script manually fails with below (on secondary). This is the same what we get from the UI.

Connecting to sqlserver-1…
Msg 3059, Level 16, State 2, Line 29
This BACKUP or RESTORE command is not supported on a database mirror or secondary replica.
Msg 3013, Level 16, State 1, Line 29
RESTORE LOG is terminating abnormally.
Disconnecting connection from sqlserver-1…
Connecting to sqlserver-1…
Msg 41145, Level 16, State 1, Line 67
Cannot join database ‘DWConfiguration’ to availability group ‘XAG’. The database has already joined the availability group. This is an informational message. No user action is required.
Disconnecting connection from sqlserver-1…

When I compared the script generated from SSMS for an old and new version, I found the difference which explained the issue clearly. This was the extra command in the new version of SSMS.

ALTER AVAILABILITY GROUP [XAG]
MODIFY REPLICA ON N'sqlserver-1' WITH (SEEDING_MODE = MANUAL)

This means that SSMS is smart enough to change the mode to MANUAL when below option is selected.

So, to conclude, the seeding was set to AUTOMATIC for the replica. When we were using an older version of SSMS, it was doing both automatic seeding and backup/restore. This was causing restore to fail because it was seeded already and added successfully.

You can use below query to find replica seeding mode.

SELECT ag.name
	,replica_server_name
	,seeding_mode_desc
FROM sys.availability_replicas ar
	,sys.availability_groups ag
WHERE ag.group_id = ar.group_id
	AND name = 'AGSQLAGDB'

Have you seen such errors of SSMS? Please comment and let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Add Database to Availability Group Failure – This BACKUP or RESTORE Command is Not Supported on a Database Mirror or Secondary Replica

Earlier I have written few blogs about automatic seeding feature. In this blog we would learn how to fix error: Automatic seeding of availability database ‘SQLAGDB’ in availability group ‘AG’ failed with a transient error.

Here are my earlier blogs about seeding failures.

SQL SERVER – AlwaysOn Automatic Seeding – Database Stuck in Restoring State

SQL SERVER – AlwaysOn Automatic Seeding Failure – Failure_code 15 and Failure_message: VDI Client Failed

In both scenarios, the database was started to restore, and it failed in the middle due to which it was in “Restoring” state. But in my client’s situation, the database was not getting listed in the databases folder in SSMS.

Automatic Seeding and Error

Here is the way to reproduce the error

CREATE DATABASE [SQLAuthority]
ON PRIMARY
(NAME = 'SQLAuthority', FILENAME = 'C:\Temp\SQLAuthority.mdf')
LOG ON
(NAME = 'SQLAuthority_log', FILENAME = 'C:\Temp\SQLAuthority_log.ldf')
GO
ALTER DATABASE SQLAuthority ADD FILE
(NAME = 'SQLAuthority_D1', FILENAME = 'C:\Temp\D1.ndf')
GO
ALTER DATABASE SQLAuthority ADD LOG FILE
(NAME = 'SQLAuthority_L1', FILENAME = 'C:\Temp\L1.ndf'),
(NAME = 'SQLAuthority_L1', FILENAME = 'C:\Temp\L2.ndf')
GO
BACKUP DATABASE SQLAuthority TO DISK = 'SQLAuthority.bak'

This would create a database with two physical transaction log files with a same logical name. Once done, use Automatic Seeding and add a database to the availability group. It is not going to fail from the UI. Here is what you would see.

If you investigate and look at ERRORLOG file on secondary, here are the failure

• Error: 911, Severity: 16, State: 1.
• Database ‘SQLAuthority’ does not exist. Make sure that the name is entered correctly.
• Error: 4353, Severity: 16, State: 1.
• Conflicting file relocations have been specified for file ‘SQLAuthority_L1’. Only a single WITH MOVE clause should be specified for any logical file name.
• Error: 3013, Severity: 16, State: 1.
• RESTORE DATABASE is terminating abnormally.
• Automatic seeding of availability database ‘SQLAuthority’ in availability group ‘SQLAUTH-AG’ failed with a transient error. The operation will be retried.

This was a coincidence that I have written a blog about this error in a standalone scenario.

SQL SERVER – FIX: Msg 4353 – Conflicting File Relocations Have Been Specified for File. Only a Single WITH MOVE Clause Should be Specified for Any Logical File Name

I looked at the database and there were no duplicate logical names! I asked about this history of the database and if there was any file added and later removed and the answer was “Yes”.

WORKAROUND/SOLUTION

I tried a lot of things, removing files completely, breaking AG, removing the database from AG. But none worked. I was not able to add the database to AG.

Finally, I used the workaround given in my blog to rename the logical file to some new name.

USE [SQLAuthority]
GO
ALTER DATABASE [SQLAuthority] MODIFY FILE (NAME=N'SQLAuthority_L1', NEWNAME=N'SQLAuthority_L2')
GO

Once the command was executed, I was able to add the database to AG via automatic seeding. Hopefully, Microsoft would see this blog and fix duplicate logical file name issue. Until then you can use a workaround.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Automatic Seeding of Availability Database ‘SQLAGDB’ in Availability Group ‘AG’ Failed With a Transient Error. The Operation Will be Retried

One of my clients contacted me for whom I assisted in configuring Always On Availability Groups for the first time. They were getting an error whole taking log backup on secondary replica: Log backup for database “SQLAuthority” on secondary replica failed because the new backup information could not be committed on the primary database.

During implementation, I explained to them various topologies, pros, and cons and assisted them in doing the end-to-end implementation as this was their first availability group. If you want any such implementation related assistance, feel free to contact me.

Secondary Replica

After I implemented their first availability group, they got comfortable and implemented distributed availability group on their own. They contacted me and informed that things are going well but they are seeing an error when they take log backup of an availability database on a secondary replica of the secondary availability group. Below is the message from job history.

Log backup for database “SQLAuthority” on secondary replica failed because the new backup information could not be committed on the primary database. Check the database status in the SQL Server error log of the server instance that is hosting the current primary replica. If the primary database is participating in the availability group, retry the operation. Check the database status in the SQL Server error log of the server instance that is hosting the current primary replica. If the primary database is participating in the availability group, retry the operation. [SQLSTATE 42000] (Error 35296)  BACKUP LOG is terminating abnormally.

To make things simple, you can imagine a deployment like this.

myAG1 (mynode1, myNode2) <- myDistributedAG –> myAG2 (myNode3, myNode4)

Distributed availability group is created between myAG1 and myAG2. myNode1 is global primary and myNode3 is a forwarder. The failure of the job is happening on myNode4.

SOLUTION/WORKAROUND

I tried this in my lab, and I was able to get exactly the same error. I searched further and found that this is by design. I found below information in this link.

In a distributed availability group, backups can be performed on secondary replicas in the same availability group as the active primary replica, or on the primary replica of any secondary availability groups. Backups cannot be performed on a secondary replica in a secondary availability group because secondary replicas only communicate with the primary replica in their own availability group. Only replicas that communicate directly with the global primary replica can perform backup operations.

When I informed this to my client, they were able to understand this limitation and modify the scripts to take backup accordingly.

Have you seen such errors while working with the availability group? Please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Error 35296: Log Backup for Database “SQLAuthority” on Secondary Replica Failed Because the New Backup Information Could Not be Committed on Primary Database

In the past, I have written a few blogs where I explained a few issues about the failure of listener creation in the cluster. In this blog, we would see another cause of listener creation failure.

I explained that whenever there are any issues related to the cluster resources, we should always look at the cluster log. If you are not sure how to generate cluster logs, read my earlier blog on the same topic. SQL SERVER – Steps to Generate Windows Cluster Log?

• Listener Name – ProdListener
• Windows Cluster Name – WinCluster

Here were the messages in the cluster log

[RES] Network Name : AccountAD: PopulateNetnameADState – DCFlags 1073745937 LdapReferralAllowed 0
[RES] Network Name: [NNLIB] FindSuitableDCNew – objectName ProdListener, username – WinCluster$, firstChoiceDCName – \\DC01.domain.com
[RES] Network Name: [NNLIB] Found first choice DC that has the object ProdListener DCName – \\DC01.domain.com.

[RES] Network Name: [NNLIB] GetOUForVCOCreation – OU name of CNO is OU=IN,DC=Domain,DC=Com
[RES] Network Name: [NNLIB] GetOUForVCOCreation – CNO SID is S-1-5-21-2987268983-2468375096-37832846-293957
[RES] Network Name : AccountAD: Object ProdListener already exists, using DC: \\DC01.domain.com

[RES] Network Name: [NNLIB] Changing Password for object ProdListener on DC: \\DC01.domain.com result: 5
[RES] Network Name: [NN] IdentityLocal End Impersonating
[RES] Network Name : AccountAD: Provisioning computer object failed with error 5
[RES] Network Name : AccountAD: OnInitializeEnd: 5
[RES] Network Name : Configuration: InitializeReplyCreation of NetName (type Singleton), result: 5, IsCanceled: false

I have trimmed the cluster logs and kept only interesting messages. In my client’s case, they already had the Listener (ProdListener) provisioned in the domain controller. The most relevant error is “Changing Password for object ProdListener on DC: \\DC01.domain.com result: 5”. This essentially means that the Cluster is trying to change the password of the object and failing with access denied. The key part here is that this operation is done by “username – WinCluster$”. This is not a domain user but a cluster computer object, also known as CNO.

“Access is denied” error can be caused due to many other reasons which can cause network name or client access point resource in cluster creation to fail. Based on my experience one of the most common causes would be where the Domain Administrator does not allow the CNO “Read All Properties” and “Create Computer Objects” permissions. You might see “Access is denied” in the event log.

WORKAROUND/SOLUTION – Listener Failure

Based on the cluster log snipper, it was clear that a domain admin needs to provide access on VCO to CNO. VCO is a virtual computer object which is a listener in Always On scenario. CNO is a cluster name object which is a computer object for Windows Cluster network name in active directory.

• Open the Active Directory Users and Computers Snap-in (dsa.msc) on a domain controller.
• In Menu > View -> check Advanced Features. (Otherwise, we would not see option explained in next steps)
• Right-click the OU/Container where we want the VCO (ProdListener in above example) to be created and click “New” -> “Computer”
• Provide a name for the object (This will be Listener Name for AG) and click “OK”.
• Right-click on the VCO which we just created and select “Properties”. Click the Security tab and then click “Add”:
• Enter the CNO (Make sure to select “Computers” option in the “Object Types” window) and click “OK”. The CNO is a Cluster Name Object. This is the name of the Windows Cluster name NOT listener or FCI name.
• Give CNO “Full Control” over the VCO.

After this, we should be able to bring listeners online in the cluster manager.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Always On Listener Failure – Provisioning Computer Object Failed With Error 5

In the past, I have written a few blogs where I explained a few issues about the failure of listener creation Failure in a cluster. In this blog, we would see another cause of listener creation failure.

Whenever I get issues while creating a listener in SQL Server, I look at the failover cluster manager to figure out the resource which is failing and then I generate a cluster log. If you are not sure how to generate cluster logs, read my earlier blog on the same topic. SQL SERVER – Steps to Generate Windows Cluster Log?

Here is the setup of my lab. This would help you in co-relating the cluster log from your environment.

• Listener Name – ProdListener
• Windows Cluster Name – WinCluster

As I mentioned my listener was failing, which is a client access point in the cluster manager. In the below lines from the cluster log, I have removed the timestamp to avoid clutter.  Here is the trimmed version of the cluster log from the node where the network name resource was not coming online.

INFO [RES] Network Name : AccountAD: Initializing Name: ProdListener, NetbiosName: ProdListener, Type: Singleton, Created: false
INFO [RES] Network Name : AccountAD: PopulateNetnameADState – DCFlags 1073745937 LdapReferralAllowed 0
INFO [RES] Network Name: [NNLIB] FindSuitableDCNew – objectName ProdListener, username – WinCluster$, firstChoiceDCName – \\domain.com
INFO [RES] Network Name: [NNLIB] Found first choice DC that has the object ProdListener DCName – \\domain.com.
INFO [RES] Network Name: [NNLIB] GetOUForVCOCreation – OU name of CNO is OU=IN,DC=Domain,DC=Com
INFO [RES] Network Name: [NNLIB] GetOUForVCOCreation – CNO SID is S-1-5-21-2987268983-2468375096-37832846-293957
INFO [RES] Network Name : AccountAD: Object ProdListener already exists, using DC: \\domain.com
INFO [RES] Network Name: [NNLIB] Trying to enable disabled object ProdListener on DC: \\domain.com (Flags 0000000000001022)
WARN [RES] Network Name : AccountAD: Enabling object ProdListener failed with error 5

In above cluster log, we can see INFO, WARN, ERR which is similar to what we see in windows event logs, Warning, Information and Error. The most relevant error is “Network Name <ProdListener>: AccountAD: Enabling object ProdListener failed with error 5”. This essentially means that Cluster is trying to enable the object and failing with access denied error. They key part here is that this operation is done by “username – WinCluster$”. This is not a domain user but a cluster computer object, also known as CNO.

“Access is denied” error can be caused due to many other reasons which can cause network name or client access point resource in cluster creation to fail. Based on my experience one of the most common causes would be where the Domain Administrator does not allow the CNO “Read All Properties” and “Create Computer Objects” permissions. You might see “Access is denied” in the event log.

WORKAROUND/SOLUTION – Listener Creation

Based on the cluster log snippet, it was clear that a domain admin needs to provide access on VCO to CNO. VCO is a virtual computer object which is a listener in Always On scenario. CNO is a cluster name object which is a computer object for Windows Cluster network name in active directory.

• Open the Active Directory Users and Computers Snap-in (dsa.msc) on a domain controller.
• In Menu > View -> check Advanced Features. (Otherwise, we would not see option explained in next steps)
• Right-click the OU/Container where we want the VCO (ProdListener in above example) to be created and click “New” -> “Computer”
• Provide a name for the object (This will be Listener Name for AG) and click “OK”.
• Right-click on the VCO which we just created and select “Properties”. Click the Security tab and then click “Add”:
• Enter the CNO (Make sure to select “Computers” option in the “Object Types” window) and click “OK”. The CNO is a Cluster Name Object. This is the name of the Windows Cluster name NOT listener or FCI name.
• Give CNO “Full Control” over the VCO.

After this, we should be able to bring VCO online in the cluster manager. Listener creation also worked and it was a happy ending.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Always On Listener Creation Failure – Enabling Object ProdListener Failed With Error 5

As a part of my work, I also provide consultancy to fix the issues. It has three advantages, first one is that I earn money and the second one, more important one, is that I get to troubleshoot and learn, and the third one is I share it further via my blog. In this blog, I would share my learning about the situation where my client was not able to add a database in an availability group and it was failing with error: This database is encrypted by database master key, you need to provide the valid password when adding it to the availability group.

My client said that he can’t add a database to an existing Availability Group. It kept on asking for a password, but the database doesn’t have TDE enabled. He also checked and found that no encryption used. Interestingly, this database as part of an availability group and due to some reason, they have to take it out and trying to add it back. The error message is as follows:

Here is the text of error message:

This database is encrypted by database master key, you need to provide valid password when adding it to the availability group.

Whenever I have such a situation, I always use a profiler to understand what the application is doing to get the details. In the profiler, I could see the below query.

SELECT
c.create_date AS [CreateDate]
FROM
sys.symmetric_keys AS c
WHERE
(c.symmetric_key_id = 101)

When I executed the same query in SSMS, we got a row for this database.

The name of this key was ##MS_DatabaseMasterKey## which means that there was a database master key created in the database, which is causing this issue.

WORKAROUND/SOLUTION – Database Master Key

First, make sure that the database is not having TDE enabled and they are no other encryption used by this database. Here is the command which we used to fix the issue.

Use [DatabaseName]
GO
Drop Master Key

After this, they were able to add the database to AG.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Always On Error: This Database is Encrypted by Database Master Key, You Need to Provide Valid Password When Adding it to the Availability Group

SQL Always-On feature is so tied to cluster that sometimes you would need to look at various logs to fix a failure. In this blog I would share my learning about fixing error LogonUserExEx fails for user & GetToken – Logging on as the CNO failed with error 1326.

Let me explain the situation and the steps I took to fix the error. The issue was that the listener network name resource was not coming online in the cluster manager. When I checked the properties of the resource, I found the below messages.

On the properties window, I have highlighted these messages.

DNS Status: The handle is invalid.

Kerberos: The user name or password is incorrect.

I asked my client to generate a cluster log. Here is the article which you can follow to generate the logs. SQL SERVER – Steps to Generate Windows Cluster Log?

In the cluster log, the relevant lines are below.

INFO [RES] Network Name: [NNLIB] PopulateKerbKDCLookupCache – DC flags 0
INFO [RES] Network Name: [NNLIB] LsaCallAuthenticationPackage success with a request of size 96, result size 0 (status: 0, subStatus: 0)
INFO [RES] Network Name: [NNLIB] Priming local KDC cache to \\dc.domain.com for domain label ad
INFO [RES] Network Name: [NNLIB] LsaCallAuthenticationPackage success with a request of size 82, result size 0 (status: 0, subStatus: 0)
WARN [RES] Network Name: [NNLIB] LogonUserExEx fails for user SQLAUTHCLU01$: 1326 (useSecondaryPassword: 0)
WARN [RES] Network Name: [NNLIB] LogonUserExEx fails for user SQLAUTHCLU01$: 1326 (useSecondaryPassword: 1)
INFO [RES] Network Name: [NNLIB] Logon failed for user SQLAUTHCLU01$ (Error 1326), DC \\dc.domain.com, domain domain.com
ERR [RES] Network Name: [NN] GetToken – Logging on as the CNO failed with error 1326
ERR [RES] Network Name : Dns: Obtaining token threw exception, error 6
ERR [RES] Network Name : Dns: Failed DNS registration with error 6 for Name: SQLAUTHCLU01 (Type: Singleton)

We can see two errors in above log, 1326 and 6. The meaning of those is matching with what was shown in the UI.

WORKAROUND/SOLUTION – GetToken

I search at various places on the internet and found that this can be fixed by repair of the computer objects. Below is the screenshot for that.

After that, I also asked them to restart the nodes and then the listener came online.

Have you seen such errors in cluster logs?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – LogonUserExEx Fails for User & GetToken – Logging on as the CNO Failed With Error 1326

One of my clients enabled query store features to perform some troubleshooting by themselves. Since they were not able to find the cause and fix the issue, the contacted me via my famous service Comprehensive Database Performance Health Check. I was able to fix the issue for them and later they wanted to disable query store feature. They contacted me again not to fix perf issues but because they were not able to disable query store features for their database. In this blog, I would share my finding of cause of unable to disable Query Store for Always On Database.

Here is the command which they executed

ALTER DATABASE [PROD_DB]
SET QUERY_STORE = OFF

This query was getting blocked with background threads. These were the values in the command column in sys.dm_exec_requests for those threads.

• QUERY STORE APRC CHECK
• QUERY STORE BACKAGOUND FLUSH DB
• QUERY STORE ASYNC FLUSH

Here was their setting of query store for this database.

I then check “DBCC OPENTRAN” for this database and found one transaction. This SPID was the same SPID which was a background task doing “QUERY STORE BACKAGOUND FLUSH DB”

Transaction information for database ‘PROD_DB’.
Oldest active transaction:
SPID (server process ID): 54s
UID (user ID) : -1
Name : QDS nested transaction
LSN : (1302919:41223:132)
Start time : Mar 26 2020 1:18:49:863PM
SID : 0x0
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

I suspected that this was due to QDS processing its async captures.

WORKAROUND/SOLUTION – Disable Query Store

I searched for Microsoft documentation which might help in query store data flush. I found trace flags about the query store. They are 7745 and 7752.

Here were the steps are done to fix the issue for them.

1. Enabled both trace flags as startup parameters on primary and secondary. No restart yet.
2. Restarted SQL Service on Node2 (secondary)
3. Performed failover of availability group from Node1 to Node2.
4. Executed ALTER command to disable Query Store.
5. Restart SQL Service on Node1.
6. Failed back availability group to Node1 (they wanted AG is run on this node)

By following the above steps, we were able to disable the query store for the database. Have you encountered such a situation? Please comment and share the steps which you have followed.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Disable Query Store for Always On Database

Question: What is the Transactional Replication Supported Version Matrix?

Answer: Most of the projects I do are about Comprehensive Database Performance Health Check. When we start a new project we have limited data and our workflow is not intense. However, I often like to get involved with projects which are not related to performance tuning when I have enjoyed working with them earlier.

Recently, while working on the health check project, one of the realizations which we had with our client was that we need to have another instance of our server where we can forward our read requests. Now the ideal solution was to configure AlwaysOn but that was not something would work out due to licensing issues of the version which the client had available. Additionally, the need for the data was near real-time hence we can’t do the traditional backup/restore or log shipping.

After carefully thinking we decided to opt for the replication. When we started to work with the replication the very first question raised was about the supported version for the transactional replication.

The rules for the supported versions are pretty simple and clearly written on the Replication Backward Compatibility.

The rules for transactional replications are:

• A Distributor can be any version as long as it is greater than or equal to the Publisher version
• A Publisher can be any version as long as it less than or equal to the Distributor version
• A Subscriber to a transactional publication can be any version within two versions of the Publisher version

In short:

• Publisher version <= Distributor version
• Subscriber (two versions) => Publisher or Subscriber (two versions) <= Publisher

For example, our client was using SQL Server 2012 for a publisher. We can use either SQL Server 2012, 2014, 2016, 2017 or 2019 as a distributor and SQL Server 2008/R2, 2012, 2014, or 2016 as a subscriber. You can see the complete matrix over the Transactional Replication Matrix.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on What is Transactional Replication Supported Version Matrix? – Interview Question of the Week #274

It is not possible to Propagation of Compatibility Level in AlwaysOn or mirroring from primary to secondary mirroring. Well, that’s what I wanted to share in this blog post.

I often get the question in Comprehensive Database Performance Health Check about Propagation of Compatibility Level in AlwaysOn. Let us discuss that today. Let us say that if you have a primary server where you are modifying compatibility level, will that automatically reflect on the secondary server. The answer is no, it is not going to happen.

If you want to change the compatibility level of your secondary server, you will have to failover your workload from a primary server to the secondary server and change your compatibility on the secondary server, and later on failback if you want to use your other server.

Here are my few recent videos and I would like to know what is your feedback about them. You can subscribe to my youtube channel here.

• Forwarded Records and Performance – SQL in Sixty Seconds #155
• Hide Code in SSMS – SQL in Sixty Seconds #154
• Zoom in SSMS – SQL in Sixty Seconds #153
• Transfer Schema of Table – SQL in Sixty Seconds #152
• Find a Table in Execution Plan – SQL in Sixty Seconds #151

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – AlwaysOn and Propagation of Compatibility Level

How to Find Instance Name for Availability Group Listener? I was recently asked this by DBA at  Comprehensive Database Performance Health Check. Well, the question is very valid if you are familiar with how Availability Group Listener works.

If you are connected to an Availability Group listener, the name displayed in Object Explorer and Properties may be different from what your instance name is actually. In this case, you may have to just use the T-SQL for the same.

Here is the T-SQL for the same.

SELECT @@Servername

Once you run the command above, it will give you a different server name from the Availability Group Listener name. Well, that’s it for today. What do you think of this small but very important blog post?

If you liked this blog, please do not forget to subscribe to my YouTube Channel – SQL in Sixty Seconds.

Here are my few recent videos and I would like to know what is your feedback about them.

• Read Only Tables – Is it Possible? – SQL in Sixty Seconds #179
• One Scan for 3 Count Sum – SQL in Sixty Seconds #178
• SUM(1) vs COUNT(1) Performance Battle – SQL in Sixty Seconds #177
• COUNT(*) and COUNT(1): Performance Battle – SQL in Sixty Seconds #176
• COUNT(*) and Index – SQL in Sixty Seconds #175
• Index Scans – Good or Bad? – SQL in Sixty Seconds #174
• Optimize for Ad Hoc Workloads – SQL in Sixty Seconds #173
• Avoid Join Hints – SQL in Sixty Seconds #172
• One Query Many Plans – SQL in Sixty Seconds #171

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Find Instance Name for Availability Group Listener

SQL Server 2022 introduces an intriguing feature – the Query Store for secondary replicas. This feature extends the Query Store functionality from primary replicas to secondary replica workloads. In this post, we explore this feature, how to enable it, and some considerations regarding its usage. This feature is currently in preview, meaning it’s not yet intended for production deployments.

What is the Query Store for Secondary Replicas?

The Query Store for secondary replicas feature allows secondary replicas to send query execution information back to the primary replica, which would typically be stored in the Query Store. The primary replica then persists this data within its own Query Store. This results in a shared Query Store between the primary and all secondary replicas. The Query Store exists on the primary replica and stores data for all replicas together.

Enabling the Query Store for Secondary Replicas

Before enabling the Query Store for secondary replicas, you must enable trace flag 12606. Here’s how you can enable the trace flags:

• Launch SQL Server Configuration Manager in Windows.
• Right-click on the SQL Server instance service for your SQL Server 2022 (16.x) instance in the list of SQL Server Services and select Properties.
• Select the Start Parameters tab and add the values: -T12606.

Remember, you must restart the SQL Server instance service for these changes to take effect.

Setting up the Query Store for Secondary Replicas

Firstly, you must have an Always On availability group on your SQL Server instance. If Query Store is not already enabled and in READ_WRITE mode on the primary replica, you must enable it with the following commands:

ALTER DATABASE [DBName] SET QUERY_STORE = ON;
GO
ALTER DATABASE [DBName] SET QUERY_STORE
( OPERATION_MODE = READ_WRITE );

To enable the Query Store on all secondary replicas, connect to the primary replica and execute the following:

ALTER DATABASE [DBName]
FOR SECONDARY SET QUERY_STORE = ON (OPERATION_MODE = READ_WRITE);
GO

To disable the Query Store on all secondary replicas, you can use a similar command:

ALTER DATABASE [DBName]
FOR SECONDARY SET QUERY_STORE = OFF;
GO

Conclusion

We must consider the performance implications as we delve into using the Query Store for secondary replicas. The same channel is used by secondary replicas to send query information back to the primary replica and to keep secondary replicas updated. This can result in the Query Store size growing and potential slowdowns during heavy system loads.

Despite these considerations, the Query Store for secondary replicas in SQL Server 2022 paves the way for more efficient data sharing between primary and secondary replicas. While this feature is not yet ready for production deployments, understanding its workings and performance implications is invaluable for database administrators and developers preparing for future advancements in SQL Server technology.

Before you test this feature on your development server, I recommend reading the official documentation. Any updates or changes will be documented there first.

You can always reach out to me on Twitter.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Query Store Feature for Secondary Replicas