Once, while doing a load testing with my client, I was shown below error while taking log backups on secondary replica.

Msg 35295, Level 16, State 1, Line 1
Log backup for database “MyAppDB” on a secondary replica failed because the last backup LSN (0x0016ff11:00032258:0001) from the primary database is greater than the current local redo LSN (0x0016ffec:0006ee28:0001). No log records need to be backed up at this time. Retry the log-backup operation later.
Msg 3013, Level 16, State 1, Line 1
BACKUP LOG is terminating abnormally.

This error was interesting because as per documentation we can take backup on the secondary replica in AlwaysOn configuration. I quickly checked AlwaysOn Dashboard from Management studio, all were looking green and databases were synchronized. Then I checked transaction log file (LDF) size and it was very big which made me think in different directions. Went to dashboard again and added Redo queue size. We found that REDO thread was blocked by a user query. As per Sys.dm_exec_requests we found that SPID 28 (which was showing DB STARTUP) was blocked by a user SPID 255. Then we used conventional way and found that SPID 255 was running SELECT INTO query for many hours. This was from a job which had a complex join with huge tables.

WORKAROUND/SOLUTION

If you find the same error while taking log backups on secondary replica, then check REDO THREAD and check if its blocked. Based on criticality you need to take a decision to KILL the SPID which is blocking system SPID.

As soon as we killed user query, the “Recovery Queue” for that database started coming down. Once it came down to zero, we could take a transaction log backup on the secondary.

Have you seen REDO blocked in production environment? Do you monitor this by monitoring tool?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX : Msg 35295, Level 16 – Log Backup For Database ‘DBName’ on a Secondary Replica Failed

As a part of my passion, I try to help people in fixing simple issues and explain the behavior by looking at logs. Here is an email from my friend about Availability Group Missing or Deleted Automatically.

Pinal,
We need your services again to help us understand the cause of a strange issue.

We had a network issue earlier today. After that we observed that there were two availability groups that were not only down, but also no longer even listed under the Availability Groups in SQL Server Management Studio. When we checked failover cluster manager, they do show up under Services and Applications in the Failover Cluster Manager but in failed states. It is interesting that on the node that has missing AGs, the databases show up as “Restoring”

Just to make sure it’s not a ghost effect, we have tried SQL Server service restarts and reboots but the fault is unchanged.

Any quick ideas?

Thanks,
John

Since this was an interesting issue, I asked John to share the SQL Server ERRORLOG SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

I looked into ERRORLOG and found below interesting message on the replica where AG was missing.

2017-02-26 13:26:43.08 spid38s Error: 19435, Severity: 16, State: 1.
2017-02-26 13:26:43.08 spid38s AlwaysOn: WSFC AG integrity check failed for AG ‘PROD_AG’ with error 41041, severity 16, state 0.
2017-02-26 13:26:43.08 spid38s AlwaysOn: The local replica of availability group ‘PROD_AG’ is being removed. The instance of SQL Server failed to validate the integrity of the availability group configuration in the Windows Server Failover Clustering (WSFC) store. This is expected if the availability group has been removed from another instance of SQL Server. This is an informational message only. No user action is required.
2017-02-26 13:26:43.08 spid38s The state of the local availability replica in availability group ‘PROD_AG’ has changed from ‘SECONDARY_NORMAL’ to ‘NOT_AVAILABLE’. The replica state changed because of either a startup, a failover, a communication issue, or a cluster error. For more information, see the availability group dashboard, SQL Server error log, Windows Server Failover Cluster management console or Windows Server Failover Cluster log.

CONCLUSION

From the above message, I think, SQL was not able to communicate with Windows cluster and as a safe guard the availability group was removed from SQL.

If this happens, there is no data loss. We just need to recover databases, if needed and drop AG completely. Once cleanup is successful, it should be easy to reconfigure availability group.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Availability Group Missing or Deleted Automatically?

I can tell you how many times I have heard about this error. This can appear in any of the situation where SQL needs to create a cluster network name resource in the WSFC Cluster. Here are the two situations I can think of:

1. Installing SQL Server Failover Cluster instance.
2. Creating listener in the AlwaysOn availability group.

Below is the error which was sent by one of my clients. This appeared while creating listener from management studio.

Here is the text of the message.

The WSFC cluster could not bring the Network Name resource with DNS name ‘<DNS name>’ online. The DNS name may have been taken or have a conflict with existing name services, or the WSFC cluster service may not be running or may be inaccessible. Use a different DNS name to resolve name conflicts, or check the WSFC cluster log for more information.

The attempt to create the network name and IP address for the listener failed. The WSFC service may not be running or may be inaccessible in its current state, or the values provided for the network name and IP address may be incorrect. Check the state of the WSFC cluster and validate the network name and IP address with the network administrator.

Above error can be caused due to many reasons which can cause network name or client access point resource in cluster creation to fail. One of the most common cause would be where the Domain Administrator does not allow the CNO “Read All Properties” and “Create computer Objects” permissions. You might see “Access is denied” in the event log.

Here are the steps, which are also known as prestaging of virtual computer object (VCO) in domain controller.

• If possible, connect to domain controller. Ensure that we are logged in as a user that has permissions to create computer objects in the domain.
• Open the Active Directory Users and Computers Snap-in (dsa.msc).
• In Menu > View -> Advanced Features. (Otherwise, we would not see option explained in next steps)
• Right click the OU/Container where we want the VCO to be created and click “New” -> “Computer”
• Provide a name for the object (This will be your SQL Server Network Name in FCI or Listener Name in AG) and click “OK”:
• Right click on the on the VCO which we just created and select “Properties”. Click the security tab and then click “Add”:
• Enter the CNO (Make sure to select “Computers” option in the “Object Types” window) and click “OK”. The CNO is a Cluster Name Object. This is the name of the Windows Cluster name NOT listener or FCI name.
• Give CNO “Full Control” over the VCO.

If all above steps are followed, we should not get access denied and if we try creating Listener, it should be successful.

What are the other errors you have seen while creating listener?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – AlwaysOn Listener Error – The WSFC Cluster Could Not Bring the Network Name Resource With DNS Name ‘DNS name’ Online

This is one of the common errors I have seen while working with customers who are using SQL Server AlwaysOn availability groups. Once this error comes, the resource in the cluster goes to failed state and in SQL Server Management Studio, we should be an availability group in resolving state. Resolving state essentially means that the role of availability group is neither primary nor secondary.

Here is the snippet from ERRORLOG when the lease expires. I have tried to explain the meaning of each line.

2017-02-27 19:31:07.34 Server Error: 19407, Severity: 16, State: 1.
2017-02-27 19:31:07.34 Server The lease between availability group ‘PRODAG’ and the Windows Server Failover Cluster has expired. A connectivity issue occurred between the instance of SQL Server and the Windows Server Failover Cluster. To determine whether the availability group is failing over correctly, check the corresponding availability group resource in the Windows Server Failover Cluster.

Above message means the lease between windows cluster and SQL Server.

2017-02-27 19:31:07.34 Server      AlwaysOn: The local replica of availability group ‘PRODAG’ is going offline because either the lease expired or lease renewal failed. This is an informational message only. No user action is required.

Due to lease renewal failure, the AG resource in the cluster would go to failed state.

2017-02-27 19:31:07.34 Server      The state of the local availability replica in availability group ‘PRODAG’ has changed from ‘PRIMARY_NORMAL’ to ‘RESOLVING_NORMAL’.  The state changed because the lease between the local availability replica and Windows Server Failover Clustering (WSFC) has expired.  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.

Since AG is failed, the AG state in SQL Server would change from PRIMARY to RESOLVING_NORMAL. At the same time, on secondary, we should see state from SECONDARY_NORMAL to RESOLVING_NORMAL

Now the next challenge would be to find WHY lease was expired.

SOLUTION/WORKAROUND

Based on my research on the internet, I found that in most of the cases, the lease gets expired due to shortage of resources on the machine. You can think of this as a “momentarily hang” of windows operations. Generally, we should look at the cause of slowness. The client with whom I worked, I could see tons of IO related messages like below.

2017-02-27 19:23:26.09 spid35s     SQL Server has encountered 244 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [M:\MSSQL\SAM_PRODUCTION.MDF] in database id 21.  The OS file handle is 0x0000000000001B44.  The offset of the latest long I/O is: 0x00004fcfb76000

I have explained above issue in below blog SQL SERVER – WARNING – SQL Server Has Encountered N Occurrence(s) of I/O Requests Taking Longer Than 15 Seconds

We also saw below, which again points to IO slowness.

2017-02-27 19:31:05.09 spid18s                 average writes per second: 1225.28 writes/sec
average throughput:  86.19 MB/sec, I/O saturation: 149511, context switches 190629
2017-02-27 19:31:05.09 spid18s                 last target outstanding: 278, avgWriteLatency 38

Till you find the actual cause, you can increase LeaseTimeout value so that the AG is remaining healthy

But remember that we have not fixed the issue, but applied band-aid.

Here are few more things to do.

• Limit Max Server Memory of SQL Server, if not capped.
• Consult your storage team for storage performance issues, since we see many IO stalled messages.
• Enable lock pages in memory. This will prevent work set trimming and prevent it from being paged out.  Please refer to the below link https://msdn.microsoft.com/en-IN/library/ms190730.aspx

Have you found something more than above?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – The Lease between Availability Group ‘PRODAG’ and the Windows Server Failover Cluster has Expired

Along with my performance consulting, I also assist many clients in fixing some quick issue. You can refer for various consulting services which I provide. This blog is an outcome of one such engagement where the client has deployed standalone SQL Server on two machines, created windows cluster between them and wanted to then use AlwaysOn availability group feature. This blog is about Primary Replica not being active.

Here is the text of the message (for search engines)

Joining database on secondary replica resulted in an error. (Microsoft.SqlServer.Management.HadrTasks)
——————————
ADDITIONAL INFORMATION:
Failed to join the database ‘SQLAuthority’ to the availability group ‘ProdAG’ on the availability replica ‘SRV2’. (Microsoft.SqlServer.Smo)
——————————
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
——————————
The connection to the primary replica is not active. The command cannot be processed. (Microsoft SQL Server, Error: 35250)
——————————

We get the same error even if we try to run JOIN command via T-SQL.

Msg 35250, Level 16, State 7, Line 1
The connection to the primary replica is not active. The command cannot be processed.

We checked ERRORLOG on both the replicas and found below the messages

Database Mirroring login attempt by user ‘DomainName\svc_sql.’ failed with error: ‘Connection handshake failed. The login ”DomainName\svc_sql’ does not have CONNECT permission on the endpoint. State 84.’. [CLIENT: 192.168.1.11]

SOLUTION/WORKAROUND

The error message is clear that the service account is not able to connect to SQL Server on database mirroring / AlwaysOn AG port. To fix the issue we need to run below command

GRANT CONNECT ON ENDPOINT::Hadr_endpoint TO [DomainName\svc_sql];

Once above was done, the issue was resolved and we were able to join the database to AG.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX: Msg 35250, Level 16, State 7 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

Along with performance tuning, I do provide consultancy services for deployments, including AlwaysOn availability group deployments. One of my clients who had a multi GB database wanted to configure AlwaysOn availability group and add a replica for disaster recovery purpose as well. They already had a secondary replica in the same data center for high availability purpose. Let us learn about AlwaysOn Join Error.

Since its not advisable to use UI for such big database to initialize the secondary, they were initializing secondary by will manual backup / restore of backup from primary followed by “join” option in the wizard. Whenever they tried this, below was the error.

Msg 1408, Level 16, State 211
The remote copy of database “SQLAUTH” is not recovered far enough to enable database mirroring or to join it to the availability group. You need to apply missing log records to the remote database by restoring the current log backups from the principal/primary database.

They contacted me because they were sure that there are no backups taken on primary replica, even then, somehow, they were seeing the error 1408 while trying to join the database to AG. They knew that the error is reported when there is a log backup taken from primary replica and there is a mismatch of LSN between the restored copy of secondary and current copy on the primary.

WORKAROUND/SOLUTION

While they knew that the error is due to the LSN mismatch, they missed one of the core concepts of backup taken from the AlwaysOn availability database.

It was easy to explain and demonstrate that they error would appear when a log backup is taken, after the full back-up from primary, even on a secondary replica. I used my below blog to find the history of the backup of the database.

When we ran the same query on the current secondary replica (remember, they already had one secondary) we found that log backups are taken regularly on that replica. As soon as we restored all the pending log backups to new secondary and tried join option again – it worked!

So whenever you see such errors, always use the script from my blog and find out the log backups which are taken. These log backups can happen even on secondary replica and would still maintain the chain with another log backup on other replicas.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – AlwaysOn Join Error – Msg 1408, Level 16 – The Remote Copy of Database “SQLAUTH” is Not Recovered Far Enough to Enable Database Mirroring or to Join it to the Availability Group

In SQL Server 2012 onwards, the cluster health check detection logic has been enhanced. Instead of the traditional pull mechanism of the cluster (IsAlive and LooksAlive), SQL Server (version 2012 onwards) uses a push mechanism to detect the health of the SQL instance. This is done by special stored procedure called sp_server_diagnostics. We should remember that the failover mechanism for the AlwaysOn FCI and the AlwaysOn Availability Groups is same.

While troubleshooting, it is very important to know which log to look at along with the basics of a feature. Recently, one of my clients was having trouble in bringing SQL Server AlwaysOn availability group resource online.

Whenever I am stuck with a cluster related issue, I always look at cluster log. You can refer my previous blog about how to generate cluster logs. SQL SERVER – Steps to Generate Windows Cluster Log?

Now let us change see the error logs.

Now let us inspect them heavily and you will find following error.

The User Does Not Have Permission to Perform this Action. (297)

I have removed date-time column from the output to provide the clarity. If we look at series of messages, you would notice that cluster has made a connection to SQL Server. After this it executes below statement

exec sp_server_diagnostics 10

As we can see in the next line that this execution failed with error.

The user does not have permission to perform this action

Due to above error diagnostic health check failed and SQL Server will not be able to come online in a cluster. Same error can also appear in the AlwaysOn availability group as well.

Now the real question is which user? And how to fix this issue?

WORKAROUND/SOLIUTION

The account which is used to connect to SQL Server from a cluster is a local system account. My client informed that due to hardening they have modified default permissions in SQL Server.

To fix this issue, we can add VIEW SERVER STATE permission to the SYSTEM account.

use [master]
GO
GRANT VIEW SERVER STATE TO [NT AUTHORITY\SYSTEM]
GO

Once done, the issue was resolved and SQL came online in the cluster as well.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – sp_server_diagnostics – The User Does Not Have Permission to Perform this Action. (297)

This was one of an interesting question which I heard from one of my clients who deployed Always On availability groups. I was not able to find much documentation and clarity so did some research and found an answer. This can also be an interview question as well.

Question:

I have configured Always On availability group using the Wizard. While configuration, I have selected a shared folder in below screen.

Due to some unavoidable reasons, I must remove that share.  If I remove that share, will availability group continues to function? Is it needed like the way we have shared location in log-shipping?

Answer:

Yes, we can remove that share and it won’t have any impact on the existing availability group. The purpose of the share is to do an initial sync via backup and restore method done by Wizard interface when we use Initialize now option. One backup and restore is complete and databases are synchronized, we can remove the files or even the share itself. Log Shipping needs a share to propagate transaction log backups in a shared location which is not the case with availability.

Have you faced a similar situation before? Let me know your thoughts about this blog post via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Can I Delete Always On Availability Groups Initial Sync Folder?

Earlier I wrote a blog on same error message which you can read from below link. SQL SERVER – FIX: Msg 35250, Level 16, State 7 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

One of my blog readers emailed me that he is getting same error 35250 (The Connection to the Primary Replica is Not Active. The Command Cannot be Processed.)

But he did not see “Database Mirroring login attempt by user” in the SQL Server ERRORLOG. He told me that my blog didn’t help me.

I reached him back via email and provided few steps to check. First, I asked to run below query to get port and service account details.

SELECT tep.NAME 'Endpoint Name'
	  ,sp.NAME 'Owner'
	  ,tep.state_desc 'State'
	  ,tep.port 'Port'
FROM sys.tcp_endpoints tep
INNER JOIN sys.server_principals sp ON tep.principal_id = sp.principal_id
WHERE tep.type = 4

Here was the output and this tells me that they are using the default port for Always On availability groups.

hadr_endpoint, database_mirroring, started, 5022

As a next step, I asked him to do two more things.

1. Telnet <Primary replica> 5022 from secondary replica via command line (cmd.exe).
2. Telnet <Secondary replica> 5022 from primary replica via command line (cmd.exe).

Telnet from the primary to the secondary worked, but it failed from the secondary to the primary. He was smart enough and tried to ping the IP address, but that did not work as well.

WORKAROUND/SOLUTION

Based on our test we concluded that this was an issue with network component. My client engaged his networking experts and found the cause of ping failure and fixed it.

Have you seen such error and found some more solution?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Msg 35250 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

It is interesting to see that in the past few days I have seen a similar pattern of an issue hit by three clients. All of them contacted me for a short engagement for the same issue. As you might know, every single day I keep two slots available for On Demand (50 minutes) so here is one of their emails. Let us learn about adding node in the AlwaysOn availability group.

“Pinal -Need your urgent help On Demand! We have added Node in AlwaysOn AG and things have broken. Basically, this cluster was set up a many months ago. Now, we were attempting to a new node at our DR site to the cluster. After adding the node all of the Disk Drives on the original two servers are showing offline and will not come online.” 

When I joined a session with them, I confirmed that they have added a new node to windows cluster which broke the disks. They were using Always On availability group, there was no need of shared storage. They used Add Node in Windows Failover Cluster Manager and keep moving next, next, finish. As soon as they did that, they found that their local drives became clustered. In our case, we had standalone instances for SQL and due to that the disk should be local disk and should not show up in the cluster administrator.

WORKAROUND / SOLUTION

1. We need to remove the disk resource from Failover Cluster Manager interface. Since there is no shared storage in our case, we do not want disks to be part of a cluster.
2. Then we need to bring these drives online in Disk Management (after making sure, step # 1 is completed)

ROOT CAUSE / CAUTION

While adding a new node to Windows Cluster, make sure that we uncheck “Add all eligible Storage to the Cluster”. I don’t know why Microsoft has this checked by default when there is no shared storage.

Hope this blog would help you in saving downtime which would be caused due to one checkbox. Just a single checkbox can help adding node issue in availability group.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Caution: Adding Node in AlwaysOn Availability Group

While preparing for a demo for a client I was hit with an interesting error in AlwaysOn Availability Group backup and restore scenario. In my lab setup, I had four nodes SQLAUTH1 to SQLAUTH4 and I have always on high availability turned on SQL Server 2016. I played and did several operations with database and found that I have bloated the transaction log file to 100 GB. And now I need to shrink the transaction log on SQLAUTH1. As soon as I did that, I was welcomed with below message about database cannot be shrunk until all secondaries have moved past the point.

The log for database ‘SQLAuth’ cannot be shrunk until all secondaries have moved past the point where the log was added.

It’s worth mentioning that I did add files in the database so above message is valid. I even took transaction log backup, but no luck. File usage was still shown as 99%.

WORKAROUND/SOLUTION

I looked around at AlwaysOn Dashboard and found that one of the secondary was not getting synchronized. Since it was a lab machine I decided to delete the AG and the thing came back to normal.

But in production, if you can’t afford to delete and reconfigure Always On availability group, then you need to find the cause why secondary is not getting synchronized with primary. In theory, the transaction log files on primary would keep on growing due to secondary not synchronizing with primary.  If you find that the data movement is suspended, then you may want to resume it and find the possible cause in the SQL Server ERRORLOG file

SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

One of the possible case I can think of is a change of the service account.

SQL SERVER – FIX: Msg 35250, Level 16, State 7 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

STILL STUCK?

If you are not able to find the cause, feel free to contact me and use my consulting services. As you might know that I have been an independent consultant for a while and one of the services I provide is “On Demand (50 minutes)” service. This service is very helpful for organizations who are in need immediate help with their performance tuning issue. Though, I have set working ours for my regular clients, every single day, I keep two hours available for this offering. This way, I can make sure that anyone who urgently needs my help, can avail the same. Click here to read more about it.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: The Log for Database Cannot be Shrunk Until All Secondaries Have Moved Past the Point Where the Log was Added

I was engaged with a client for an AlwaysOn project and they had some follow-up questions. I took some time to find the answers and encountered an interesting error. I am sharing them here so that others can get benefited. They informed me that they are not able to see and modify listener properties. Let us learn about this error related to opening resource.

Initially, I shared script to get the properties of the listener via T-SQL. As you can see below, we can use catalog views.

SELECT grp.name AS [AG Name],
lis.dns_name AS [Listener DNS Name],
lis.port AS [Listener Port]
FROM sys.availability_group_listeners lis
INNER JOIN sys.availability_groups grp
ON lis.group_id = grp.group_id
ORDER BY grp.name, lis.dns_name

Here is the output.

My client came back and told that networking team has asked to change RegisterAllProvidersIP setting. We are not able to use PowerShell and getting error “An error occurred opening resource”. We are not sure what wrong with the listener in the cluster.

Get-ClusterResource AGListener | Get-ClusterParameter
PS C:\> Get-ClusterResource AGListener | Get-ClusterParameter
Get-ClusterResource : An error occurred opening resource 'AGListener'.
At line:1 char:1
+ Get-ClusterResource AGListener | Get-ClusterParameter
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : ObjectNotFound: (:) [Get-ClusterResource], ClusterCmdletException
 + FullyQualifiedErrorId : ClusterObjectNotFound,Microsoft.FailoverClusters.PowerShell.GetResourceCommand

If we look at cluster the value “AGListener” we are using seems correct, but still PowerShell thinks its incorrect. Here is the screenshot from cluster manager.

I did some more searching and found that when we create Listener in through SSMS its naming convention like AGNAME_ListenerName. This is the reason that when we run the command Get-ClusterResource for the listener, we can’t see the properties. Here are the properties of the listener resource. (Right Click)

WORKAROUND/SOLUTION

Based on above explanation, we need to use the “name” as shown in properties and the command was working as expected.

Get-ClusterResource BO1AG_AGListener | Get-ClusterParameter

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Get Listener Properties Using PowerShell – An Error Occurred Opening Resource

While doing a preparation of a demo, I encountered below error while creating the listener of the AlwaysOn availability group. I was trying this from Management Studio.

Here is the text of the error message.

The configuration changes to the availability group listener were completed, but the TCP provider of the instance of SQL Server failed to listen on the specified port [AGListener:1433]. This TCP port is already in use. Reconfigure the availability group listener, specifying an available TCP port. For information about altering an availability group listener, see the “ALTER AVAILABILITY GROUP (Transact-SQL)” topic in SQL Server Books Online. (Microsoft SQL Server, Error: 19486)

If you investigate SQL Server ERRORLOG, you should see below.

2017-09-11 11:29:34.60 spid74 Error: 19476, Severity: 16, State: 4.
2017-09-11 11:29:34.60 spid74 The attempt to create the network name and IP address for the listener failed. If this is a WSFC availability group, the WSFC service may not be running or may be inaccessible in its current state, or the values provided for the network name and IP address may be incorrect. Check the state of the WSFC cluster and validate the network name and IP address with the network administrator. Otherwise, contact your primary support provider.
2017-09-11 11:30:01.19 Server The Service Broker endpoint is in disabled or stopped state.
2017-09-11 11:30:01.19 Server Error: 26023, Severity: 16, State: 1.
2017-09-11 11:30:01.19 Server Server TCP provider failed to listen on [ 10.0.1.50 1433]. Tcp port is already in use.
2017-09-11 11:30:01.19 Server Error: 26075, Severity: 16, State: 1.
2017-09-11 11:30:01.19 Server Failed to start a listener for virtual network name ‘AGListener’. Error: 10013.
2017-09-11 11:30:01.19 Server Stopped listening on virtual network name ‘AGListener’. No user action is required.
2017-09-11 11:30:01.19 Server Error: 10800, Severity: 16, State: 1.
2017-09-11 11:30:01.19 Server The listener for the WSFC resource ‘dc977169-2387-499e-8047-3b197e7ada61’ failed to start, and returned error code 10013, ‘An attempt was made to access a socket in a way forbidden by its access permissions. ‘. For more information about this error code, see “System Error Codes” in the Windows Development Documentation.
2017-09-11 11:30:01.19 Server Error: 19452, Severity: 16, State: 1.
2017-09-11 11:30:01.19 Server The availability group listener (network name) with Windows Server Failover Clustering resource ID ‘dc977169-2387-499e-8047-3b197e7ada61’, DNS name ‘AGListener’, port 1433 failed to start with a permanent error: 10013. Verify port numbers, DNS names and other related network configuration, then retry the operation.

SOLUTION/WORKAROUND

Above confirmed that there is some other process listener on 1433 port. Then I use one of my old blog to find out which process is using that port.

SQL SERVER – Unable to Start SQL Service – Server TCP provider failed to listen on [‘any’ 1433]. Tcp port is already in use.

I found that it was another system process, so I should change my listener port. Since listener was already created all I needed was to change the port. Here is the T-SQL I have used to change the port to 2433.

USE [master]
GO
ALTER AVAILABILITY GROUP [BO1AG]
MODIFY LISTENER N'AGListener' (PORT=2433);
GO

This time, command was completed without any error. I confirmed from the Errorlog that there was no error related to the listener.

2017-09-11 11:34:35.460 Server Started listening on virtual network name ‘AGListener’. No user action is required.

Have you encountered any such error? Feel free to comment and share it with others.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – AlwaysOn Availability Group Listener – This TCP Port is Already in Use

One of my clients has configured AlwaysOn availability group and wanted to automate backup to secondary replica. They debugged and found that the function sys.fn_hadr_backup_is_preferred_replica returns 0 no matter what the backup preference is set to. Since function was returning zero, the maintenance plan was ignoring the server to take backup.

Here is the output of the function for both the nodes.

As we can see, the value is zero on both nodes. I verified settings via catalog views and DMV and all were looking perfectly fine.

WORKAROUND/SOLUTION

After studying the code of fn_hadr_backup_is_preferred_replica I was able to debug and figure out that this was because of the fact that SELECT @@SERVERNAME was returning incorrect values. After digging from the client, I would that their Wintel team configure this VM from an image that had SQL installed.

Here is my own blog which I followed to fix server name.

SQL SERVER – Fix @@ServerName Property Value When Incorrect

After putting the correct name and restarting the SQL Server service, the function started working as expected.

Let me know if you have faced ever such error in SQL Server where you have to apply such fix for your SQL Server.

Reference: Pinal Dave (https://blog.sqlauthority.com) 

First appeared on SQL SERVER – AlwaysOn Availability Group Backup fn_hadr_backup_is_preferred_replica Not Working Correctly

One of my blog readers was referring below blog SQL SERVER – Create Login with SID – Way to Synchronize Logins on Secondary Server. Let us learn in this blog post about the script to sync logins between replicas.

He sent me email asking if there is a script available to create multiple logins with SID and password? I was able to spend some time and referred various online blogs/content to come up with below script.

Please let me know if you find below script useful.

SELECT 'create login [' + sp.name + '] ' + CASE 
		WHEN sp.type IN (
				'U'
				,'G'
				)
			THEN 'from windows '
		ELSE ''
		END + 'with ' + CASE 
		WHEN sp.type = 'S'
			THEN 'password = ' + master.sys.fn_varbintohexstr(sl.password_hash) + ' hashed, ' + 'sid = ' + master.sys.fn_varbintohexstr(sl.sid) + ', check_expiration = ' + CASE 
					WHEN sl.is_expiration_checked > 0
						THEN 'ON, '
					ELSE 'OFF, '
					END + 'check_policy = ' + CASE 
					WHEN sl.is_policy_checked > 0
						THEN 'ON, '
					ELSE 'OFF, '
					END + CASE 
					WHEN sl.credential_id > 0
						THEN 'credential = ' + c.name + ', '
					ELSE ''
					END
		ELSE ''
		END + 'default_database = ' + sp.default_database_name + CASE 
		WHEN len(sp.default_language_name) > 0
			THEN ', default_language = ' + sp.default_language_name
		ELSE ''
		END
FROM sys.server_principals sp
LEFT JOIN sys.sql_logins sl ON sp.principal_id = sl.principal_id
LEFT JOIN sys.credentials c ON sl.credential_id = c.credential_id
WHERE sp.type IN (
		'S'
		,'U'
		,'G'
		)
	AND sp.name <> 'sa'
	AND sp.name NOT LIKE 'NT Authority%'
	AND sp.name NOT LIKE 'NT Service%'

How to use the script?

You need to run above script on primary replica. Once we execute this we would get a result which would have commanded to create a login. Copy the output and paste in a query window. The query needs to run against secondary server.

If you are using the Windows login than create them in every instance. In windows login case, the SID is managed by active directory, so you will be able to access in all replicas members of an availability group if the login exists in the primary replica.

Please note that user information is part of the database and it would come automatically via data synchronization.

Reference: Pinal Dave (https://blog.sqlauthority.com) 

First appeared on SQL SERVER – AlwaysOn Availability Groups: Script to Sync Logins Between Replicas?

Along with my performance consulting, I have also assisted many clients in fixing some quick issues. You can refer for various consulting services which I provide. During my consulting with various clients about Always On deployment, one of the commonly reported errors as below.  (I have changed the AG Name, Node names to be more generic). Below came when we used the wizard to create an availability group. Let us learn how to fix error related to primary replica.

Joining database on secondary replica resulted in an error. (Microsoft.SqlServer.Management.HadrTasks)
——————————
ADDITIONAL INFORMATION:
Failed to join the database ‘DB’ to the availability group ‘AG1’ on the availability replica ‘NODE2’. (Microsoft.SqlServer.Smo)
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
——————————
The connection to the primary replica is not active. The command cannot be processed. (Microsoft SQL Server, Error: 35250)

This error also comes when we try to join the replica to an availability group using below T-SQL.

ALTER DATABASE DB SET HADR AVAILABILITY GROUP = AG1

And the error is below:

Msg 35250, Level 16, State 7, Line 1
The connection to the primary replica is not active. The command cannot be processed.

MY CHECKLIST

Here is the little checklist which I have made after referring various sites on the internet.

1. Ensure that AlwaysOn endpoint, generally named as Hadr_endpoint, is not blocked by firewalls (Default port 5022). To confirm this, we should do a Telnet test from each node to another node. Sometimes you need to install “Telnet Client” on the server using “Server Manager” > Add features. If you are not able to reach to port than make sure.
   1. Endpoints are already created and listening. Check ERRORLOG for this.
   2. A firewall is allowing the port. Check Firewall for this.
2. Make sure we can resolve FDQN with IP and vice versa. Generally, a ping test can be done to achieve this.
3. Check the hosts file in “C:\Windows\System32\drivers\etc” to make sure there are no incorrect entries.
4. Make sure that the startup account of the primary server is added to all secondary server’s and Startup accounts of all secondary servers are added to primary servers. In short, startup account of each replica to be added to other replicas. I have generally seen a client using one service account across all servers *cough* *cough*
5. If the service account of SQL Server is “NT Service\” or LocalSystem account, then ensure system account (Domain\ReplicaName$) of each replica is added to other replicas.  Generally, people add it as SysAdmin *cough* *cough* but it is better to give lesser permission like below

GRANT CONNECT ON ENDPOINT::[Hadr_endpoint] TO [Domain\node1$]

In my client’s case we were getting below error in ERROLROG

Database Mirroring login attempt by user Domain\SecondaryReplica$.’ failed with error: ‘Connection handshake failed. The login ‘ Domain\SecondaryReplica$’ does not have CONNECT permission on the endpoint. State 84.’. [CLIENT: PrimaryIP]

This is covered in # 5 in above checklist. So, we created a login for other replicas in each replica and granted connect permission to fix the error. Microsoft documentation also states the same: “If any server instances that are hosting the availability replicas for an availability group run as different accounts, the login each account must be created by a master on the other server instance. Then, that login must be granted CONNECT permissions to connect to the database mirroring endpoint of that server instance.”

I hope that my checklist would help you in narrowing down the cause of the error and you should be able to fix it.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Always On Availability Group Error 35250: The Connection to the Primary Replica is Not Active

One of my clients was using Always On availability group. They were having 3 nodes Always On AG setup. For simplicity, let’s call them Node1, Node2 and Node3. The first two nodes were in one subnet and node3 was in separate subnet. Let us learn about availability group failover.

The Problem

Client informed that they are not able to fail over the AG from first subnet (node1 & node2) to the second subnet (node3). On the other hand, the failover works fine between node1 and node2 (same subnet)

Initially I thought it’s due to misconfiguration of their network, but it was looking OK. When an attempt was made to fail over the AG group, it comes to “Resolving” state on all replicas. When I observed further, a dump file was generated on node3.

Here is my earlier blog for dump related issues SQL SERVER – Too Many SQLDump Files Consuming a Lot of Disk Space. What Should You Do?

***Stack Dump being sent to xxx\SQLDump0001.txt
* *****************************************************
*
* BEGIN STACK DUMP:
* 12/14/15 09:57:47 spid 52
*
* Location: HadrFstrVnnUtils.cpp:479
* Expression: SUCCEEDED (hr)
* SPID: 52
* Process ID: 5412

Followed by below message

Error: 17066, Severity: 16, State: 1.
SQL Server Assertion: File: , line=479 Failed Assertion = ‘SUCCEEDED (hr)’. This error may be timing-related. If the error persists after rerunning the statement, use DBCC CHECKDB to check the database for structural integrity, or restart the server to ensure in-memory data structures are not corrupted.

I had no idea what HadrFstrVnnUtils is. When I broke it down the words it comes as Hadr Fstr Vnn Utils. Still, I had no clue about it. I know Hadr is the earlier name for Always On and now it looks like Fstr = File Stream. If you can find what VNN is, please share and educate me.

I asked them to check SQL Server configuration manager for setting related to Filestream and it turned out that the setting was disabled on Node3 and was enabled on Node1 and Node2.

WORKAROUND/SOLUTION

• Enable FilesStream on each replica and set ‘filestream access level’ to 2

or

• Disable on all replicas and recreate the AG.

Here is the documentation about the steps to enable / change setting of Filestream. Read the details here.

(The steps below are taken from the link)

To enable and change FILESTREAM settings

1. On the Start menu, point to All Programs, point to SQL Server 2016, point to Configuration Tools, and then click SQL Server Configuration Manager.
2. In the list of services, right-click SQL Server Services, and then click Open.
3. In the SQL Server Configuration Manager snap-in, locate the instance of SQL Server on which you want to enable FILESTREAM.
4. Right-click the instance, and then click Properties.
5. In the SQL Server Properties dialog box, click the FILESTREAM tab.
6. Select the Enable FILESTREAM for Transact-SQL access check box.
7. If you want to read and write FILESTREAM data from Windows, click Enable FILESTREAM for file I/O streaming access. Enter the name of the Windows share in the Windows Share Name box.
8. If remote clients must access the FILESTREAM data that is stored on this share, select Allow remote clients to have streaming access to FILESTREAM data.
9. Click Apply.
10. In SQL Server Management Studio, click New Query to display the Query Editor.
11. In Query Editor, enter the following Transact-SQL code:

    EXEC sp_configure filestream_access_level, 2
    RECONFIGURE

12. Click Execute.
13. Restart the SQL Server service. We have to restart the SQL Server after enabling filestream.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Always On Availability Group Failover Generating Dump. Error: 17066: SQL Server Assertion: HadrFstrVnnUtils.cpp:479

One of my clients took my assistance to deploy Always On availability group. After few months, they added a Disaster Recovery node to windows cluster and availability group also. Later, they contacted me that they are not able to move availability group to DR node. They were getting an error like below in event viewer.

Cluster resource ‘SQLAG_22.22.22.22’ of type ‘IP Address’ in clustered role ‘SQLAG’ failed.

Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet.

When I looked at their network configuration, I found that they have two networks.

But when I checked listener role, it was still having only 1 IP address.

SOLUTION/WORKAROUND

They wanted to do DR test, and they did not have any free IP on the DR network. When I asked how an application would connect? They educated me that in DR, they would be using node name to connect rather than listener name.

If above was the way they wanted, then the solution for them was to remove dependency between AG and listener via cluster manager. So, I broke the dependency and failed over AG to the second node without any error. The IP was in the failed state, but AG came online because we removed the dependency.

But in general, when we have multi-subnet AG then we should have multiple IPs for the listener as well. The number of IPs would be equal to the number of subnets.

Have you worked with multi-subnet AG? What are the issues you have faced?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER- Unable to Failover AlwaysOn Availability Group to Disaster Recovery Site

One of my clients contacted me for quick On-Demand consulting. They were working under close deadlines and delivering the solution on time was critical for their success. They contacted me and explained that they were having trouble creating a listener for their two nodes AlwaysOn Availability Group. The error message which they received from T-SQL is below. Let us learn how we can fix error related to Windows Server Failover Clustering.

Msg 41009, Level 16, State 7, Line 126
The Windows Server Failover Clustering (WSFC) resource control API returned error code 5057.  The WSFC service may not be running or may not be accessible in its current state, or the specified arguments are invalid.  For information about this error code, see “System Error Codes” in the Windows Development documentation.

Msg 19476, Level 16, State 3, Line 126
The attempt to create the network name and IP address for the listener failed. The WSFC service may not be running or may be inaccessible in its current state, or the values provided for the network name and IP address may be incorrect. Check the state of the WSFC cluster and validate the network name and IP address with the network administrator.

If we really see there are three error messages but first one is having more details. In one of my earlier blog I explained that whenever there are any issues related to cluster resource, we should always look at cluster log. If you are not sure how to generate cluster logs, read my earlier blog on the same topic.

SQL SERVER – Steps to Generate Windows Cluster Log?

We looked at cluster log to see what messages we see there at the same time. I have trimmed the log by removing date time field from the log.

The cluster log has these errors:

1. IP Address <PRODAG_162.28.55.59>: IpaValidatePrivateResProperties: IP address 162.28.55.59 was detected on the network.
2. Error 5057 from ResourceControl for resource PRODAG_162.28.55.59.

If we do NET HELPMSG for 5057, we get below

SOLUTION/WORKAROUND

From above information, we can conclude that error appears because we are trying to use an IP address for the listener which is already in use. We need to use an unused IP and that should fix the issue.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on QL SERVER – Unable to Create Listener: Msg 41009 – The Windows Server Failover Clustering (WSFC) Resource Control API Returned Error Code 5057

As a part of my consulting, I observed that there is a lot of interest in using AlwaysOn availability group. I have deployed many AlwaysOn solutions from start-to-finish. Recently I encountered an issue and in this blog, we would discuss reason of an availability database to stay in restoring mode even after using automatic seeding.

If you have not heard earlier, one of the new features in AlwaysOn which was introduced in SQL Server 2016 was “automatic seeding”. In the latest version of SSMS, we can also see the option of seeding as below.

My client used the option and their database size was huge and seeding is supposed to work faster than a backup and restore method. We started seeding and I asked them to use below query to monitor the progress of seeding.

SELECT local_database_name
 ,role_desc
 ,internal_state_desc
 ,transfer_rate_bytes_per_second
 ,transferred_size_bytes
 ,database_size_bytes
 ,start_time_utc
 ,end_time_utc
 ,estimate_time_complete_utc
 ,total_disk_io_wait_time_ms
 ,total_network_wait_time_ms
 ,is_compression_enabled
FROM sys.dm_hadr_physical_seeding_stats

They contacted me again and told that seeding is completed but the database is still shown in “Restoring” state.

When I attempted to join, I received an error

Failed to join the database ‘HiDB’ to the availability group ‘HiAG’ on the availability replica ‘(LOCAL)’. (Microsoft.SqlServer.Smo)
——————————
ADDITIONAL INFORMATION:
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
——————————
The remote copy of database “HiDB” has not been rolled forward to a point in time that is encompassed in the local copy of the database log. (Microsoft SQL Server, Error: 1412)

This error is what tells us the exact cause of the issue. It looks like there is a log backup which happened in between seeding process. I asked them to use the blog and find SQL SERVER – Get Database Backup History for a Single Database

WORKAROUND/SOLUTION

In this case, after running the script from my blog, we found that a backup job which was taking log backup on another replica every 15 minutes. We disabled the job and started seeding again. This time it worked without any issue.

Have you encountered some error during AlwaysOn AG? Please share via comments and help others.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – AlwaysOn Automatic Seeding – Database Stuck in Restoring State