FIM 2010 R2: SQL timeout on using large sets in other sets

Last week a strange error appears in my environment, I was adding the All expected rule resources set to another set to give read permissions to operational admins, so that they can see the EREs on the provisioning tab of user resources. I build a set for these operational admins containing all resources that they should have read access to, in that set I include serveral objecttypes like teams and groups and also the above sets which has large amount of members.

Here is the set I’m trying to build:

EditSetError2

I do this directly in production, and all went fine, however to don’t break my deployment through the 3 stages (dev, test, prod) I also try to make this change in my other 2 environments, but then I their I got an Postprocessing Error.

I have to say that all 3 environments running on identical physical machines, with nearly the same configuration. (Windows Server 2008 R2, SQL2008R2 and FIM 2010 R2 SP1).

Here is the error I got in portal:

EditSetError1

Error processing your request: The server was unwilling to perform the requested operation.
Reason: Unspecified.
Attributes:
Correlation Id: 1292580b-150f-4921-9beb-c8761476787e
Request Id:
Details: Request could not be dispatched.

I figured out that there are also 2 errors in FIM Eventlog:

.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at Microsoft.ResourceManagement.Data.DataAccess.DoRequestCreation(RequestType request, Guid cause, Guid requestMarker, Boolean doEvaluation, Int16 serviceId, Int16 servicePartitionId)

and this:

Requestor: urn:uuid:7fb2b853-24f0-4498-9534-4e10589723c4
Correlation Identifier: 1292580b-150f-4921-9beb-c8761476787e
Microsoft.ResourceManagement.WebServices.Exceptions.UnwillingToPerformException: Other ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at Microsoft.ResourceManagement.WebServices.RequestDispatcher.CreateRequest(UniqueIdentifier requestor, UniqueIdentifier targetIdentifier, OperationType operation, String businessJustification, List`1 requestParameters, CultureInfo locale, Boolean isChildRequest, Guid cause, Boolean doEvaluation, Nullable`1 serviceId, Nullable`1 servicePartitionId, UniqueId messageIdentifier, UniqueIdentifier requestContextIdentifier, Boolean maintenanceMode)
at Microsoft.ResourceManagement.WebServices.ResourceManagementService.Put(Message request)
--- End of inner exception stack trace ---

I found out that the request is taking more time than the SQL timeout configured in FIM Service, which is 58 seconds by default. But why only in dev and test and not in production.

Sadly I have an answer on this, since I don’t do further debugging with SQL profiler, as the following changes to the FIM Service configuration file resolves the issue.

To extend the timeout modify the FIM service configuration file Microsoft.ResourceManagement.Service.exe.config by adding the both timeout parameter:

<resourceManagementService externalHostName="myfimservice" dataReadTimeoutInSeconds="1200" dataWriteTimeoutInSeconds="1200"/>

These parameters are also documented within the config file.

I’ve tried the value 120 and 300 first but the request seems to take longer, after that I decided to use value 1200. After the request completed I could see the request takes more than 6 minutes to complete. In production it could complete within the 58 seconds timeout. Very strange behavior.

However after extending the timeout and restart FIM Service the portal will still present you a timeout but the request will complete in background, you can check this in request history. There is also a possibility to extend this timeout in portal by changing the web.config, but for me that was not necessary.

Beside getting rid of that error there are still 2 question i could not answer myself:

1. Why did that affect dev and test but not production (having in mind the environments are identical) ?

2. What takes so long to add a set to another set, even it has a large amount of members ?

If you could answer one or both of them so please leave me a comment or mail.

 

FIM 2010 R2 Hotfix Build (4.1.3479.0) released

Today, just a week after the last hotfix, Microsoft released another hotfix for Forefront Identity Manager 2010 R2, which has build 4.1.3479.0

You can download this hotfix directly from this link.

Here are the issues fixed, taken from the KB article:

Issues that are fixed in this update

This update fixes the following issues that were not previously documented in the Microsoft Knowledge Base.

Read more of this post

Correct group objects with: Dynamic group has static member

Sometimes I have warnings in the group UI of my dynamic groups, telling me that “Dynamic group has static member”. Unless you cannot add static member to dynamic groups in FIM portal by yourself, it can be flow into through the Synchronization Engine, especially if you have equal precedence on the member attribute for groups. I need this setting, because groups can be managed in FIM Portal and in Active Directory.

So if someone add a user to a group in AD which is a dynamic group in FIM Portal it, this change flows into Portal and you get this warning. I still working on a solution that this will not happen in future.

I’ve tried to create a set to catch such groups in Portal and maybe send a notification message or clean up such groups with a workflow but I have no look in creating such a set. So I ended up in PowerShell once again to do this.

Here is my script to Remove all static member from dynamic groups.

add-pssnapin FIMAutomation

$grouplist = Export-FIMConfig -only -custom "/Group[MembershipLocked = 'true' and ExplicitMember = /Person]"

If ($grouplist -eq $null) { Write-Host "There is no dynamic group with static member" ; exit }

foreach ($group in $grouplist)
{
    $memberlist=($group.ResourceManagementObject.ResourceManagementAttributes | where {$_.AttributeName -eq "ExplicitMember"}).Values

    $importObject = New-Object Microsoft.ResourceManagement.Automation.ObjectModel.ImportObject
    $importObject.ObjectType = "Group"
    $importObject.TargetObjectIdentifier = $group.ResourceManagementObject.ObjectIdentifier
    $importObject.SourceObjectIdentifier = $group.ResourceManagementObject.ObjectIdentifier
    $importObject.State = 1

    foreach ($member in $memberlist)
    {
        $importChange = New-Object Microsoft.ResourceManagement.Automation.ObjectModel.ImportChange
        $importChange.Operation = [Microsoft.ResourceManagement.Automation.ObjectModel.ImportOperation]::Delete
        $importChange.AttributeName = "ExplicitMember"
        $importChange.AttributeValue = $member.Replace("urn:uuid:","")
        $importChange.FullyResolved = 1
        $importChange.Locale = "Invariant"
        $importObject.Changes += $importChange
    }

    $importObject | Import-FIMConfig
}

Notes:

The script currently only removes person objects from static member, you can modify this on your own if you have also group objects in the ExplicitMember attribute of dynamic groups.

 

FIM 2010 R2 Hotfix Build (4.1.3469.0) released

On Oct. 6 2013 Microsoft released a new Hotfix for FIM 2010 R2, which is Built 4.1.3469.0.
You can find the documentation and download link in KB2877254.

IssueList taken from the original KB Article:

Issues that are fixed or features that are added in this update

This update fixes the following issues or adds the following features that were not previously documented in the Microsoft Knowledge Base.
Read more of this post

FIM 2010 R2: Searching for Request Details in msidmCompositeType

With Forefront Identity Manager 2010 R2 Microsoft add some performance modifications to Service and Portal. By default exports from the FIM Management Agent are batched (aggregated) up to 1000 changes. So in request details you may see only an Update to msidmCompositeType ‘’ Request which includes all the changes of maybe multiple objects in the RequestParameter attribute.

Craig has also post a short description on how to deal with this objecttype.

Since this is very good for performance on exports, it is very bad for searching changes in the request log in portal. In my environment I even see changes that seems to be only relevant to one object, holding changes to multiple objects. In this particular case the requests display name is Update to msidmCompisiteType ‘myuser’ Request, but in the RequestParameter attribute I also found changes to other objects.

While you cannot actually searching the request log for changes to a specific object, especially with the new batch object type, I’ve decided to create a little script for this in PowerShell of course.

The script is based on the FIMAutomation snap-in and also the FIM PowerShell Module from Craig. Because I missed some attribute in output of the Get-FIMRequestParameter cmdlet I have modified this a little bit.

Giving an objectType, attribute and value to search, the script will retrieve all changes to this object searching all direct/single requests and also all batched updates.

To use this script you must in addition to install the FIMPowerShellModule modify the FIMPowerShellModule.psm1 file like below. I’ve only added the Mode and Target properties to the output of the Get-FIMRequestParameter function.

$RequestParameter | foreach-Object{
    New-Object PSObject -Property @{
        Mode = ([xml]$_).RequestParameter.Mode
	Target = ([xml]$_).RequestParameter.Target
        PropertyName = ([xml]$_).RequestParameter.PropertyName
        Value = ([xml]$_).RequestParameter.Value.'#text'
        Operation = ([xml]$_).RequestParameter.Operation
     } |
     Write-Output

Here is my PowerShell Script, see also comments inside the code:

param($objectType, $attribute, $searchValue)
#objectType = The objectType of the target object you are trying to get requests for.
#attribute = The attribute of the target object you want to use for searching
#searchValue = The value of the attribute of the target object you are searching requests for
#
#ex. Get-FIMRequestDetails.ps1 -objectType "Person" -attribute "AccountName" -searchValue "pstapf"
#
#This gets all requests matching the given target object.

# Load FIMAutomation SnapIn and FIMPowershellModule (http://fimpowershellmodule.codeplex.com)
if(@(get-pssnapin | where-object {$_.Name -eq "FIMAutomation"} ).count -eq 0) {add-pssnapin FIMAutomation}
Import-Module C:\Windows\System32\WindowsPowerShell\V1.0\Modules\FIM\FIMPowerShellModule.psm1

# Check if the object you are searching requests for exists in portal and get its GUID
$filter = "/" + $objectType + "[" + $attribute + "=" + $searchValue + "]"
$searchObject = Export-FIMConfig -OnlyBaseResources -CustomConfig $filter

If ($searchObject -ne $null)
{
    $searchObjectGuid = $searchObject.ResourceManagementObject.ObjectIdentifier.Replace("urn:uuid:","")
    Write-Host "Object found:" $searchValue " with GUID:" $searchObjectGuid
}
else
{
    Write-Host "The object you are searching for does not exists in FIM Portal"
    Exit
}

# Get the aggregated requests of the object you search for
$export=@()
$filter = "/Request[Target=/msidmCompositeType[/msidmElement=/" + $objectType + "[" + $attribute + "=" + $searchValue + "]]]"
$export = Export-FIMConfig -OnlyBaseResources -CustomConfig $filter

# Get the single requests of the object you search for
$filter = "/Request[Target=/" + $objectType + "[" + $attribute + "=" + $searchValue + "]]"
$export += Export-FIMConfig -OnlyBaseResources -CustomConfig $filter
$requestlist = $export | Convert-FimExportToPSObject | Sort-Object msidmCompletedTime

# Get the RequestParameter of the object you search fo from all requests and add some requestDetails
If ($requestlist.count -gt 0)
{
    $resultItems = @()
    foreach ($requestItem in $requestList)
    {
        $resultItems += $requestItem | Get-FimRequestParameter | where { $_.Target -eq $searchObjectGuid } | ForEach-Object {
        New-Object PSObject -Property @{
            Target = $_.Target
            Operation = $_.Operation
            Mode = $_.Mode
            Attribute = $_.PropertyName
            Value = $_.Value
            RequestName = $requestItem.DisplayName
            RequestGuid = $requestItem.ObjectID.Replace("urn:uuid:","")
            CompleteTime = $requestItem.msidmCompletedTime
            Status = $requestItem.RequestStatus
            }
        }
    }
    $resultItems

}
else
{
    Write-Host "No request found for the object you searched for."
}

Using Notes:

Script output is PSObject, so you can pipe the output to many other cmdlets, like Out-GridView or Convertto-Html for example. In addition you can filter the output by date also on your own.

Out-GridView is a very neat cmdlet for this, as you can show/hide columns and also do some basic filtering, try a Status:Completed filter for example.

OutGridView

Here is also some output with ConverTo-Html with some basic CSS:

OutHtml

FIM 2010: Configuration Deployment (Part 2: Configuration Objects)

As you hopefully read in Part 1, we often had to deal with deploying configuration from one stage to another. Deploying the schema is the easier part as this mostly goes 1:1 from dev to test and finally production, but I have to deal with environment specific differences in the configuration objects like Sets, Workflows or MPRs.

In synchronization rules for example I often use fixed stings, like domain or constructing DNs for provision objects to LDAP directories and many other attributes. I also have some sets which must be different in the environments as in testing stage some more people have permissions to do something rather than in production, so I want to manage some of these sets manually and don’t want them to be deployed.

So, like in the schema deployment script I put all the example scripts together and also start the exports in parallel using PowerShell jobs. In addition after joining and calculating the delta configuration I do a search and replace of the fixed string values and even delete some objects which I don’t want to deploy. These replacements and deletes are defined in a configuration XML file.

So here is the script I use to deploy my configuration changes from one environment to the other:
(This script is currently designed to run from the target system)

param([switch]$AllLocales)
# use -AllLocales parameter to deploy schema data incl. localizations

#### Configuration Section ####
$sourceFilename="D:\Deploy\Policy_Source.xml"
$sourceServer="fim.source.com"
$destFilename="D:\Deploy\Policy_Target.xml"
$destServer="fim.target.com"

$tempDeltaFile = "D:\Deploy\Policy_DeltaTemp.xml"
$finalDeltaFile = "D:\Deploy\Policy_DeployData.xml"
$undoneFile = "D:\Deploy\undone.xml"

$config=[XML](Get-Content .\DeployReplaceConfig.xml)

#### Cleanup old data files ####
remove-item $sourceFilename -ErrorAction:SilentlyContinue
remove-item $destFilename -ErrorAction:SilentlyContinue
remove-item $tempDeltaFile -ErrorAction:SilentlyContinue
remove-item $finalDeltaFile -ErrorAction:SilentlyContinue
remove-item $undoneFile -ErrorAction:SilentlyContinue

#### Load FIMAutonation cmdlets ####
if(@(get-pssnapin | where-object {$_.Name -eq "FIMAutomation"} ).count -eq 0) {add-pssnapin FIMAutomation}

#### Function Definitions ####
function CommitPortalChanges($deployFile)
{
	$imports = ConvertTo-FIMResource -file $deployFile
	if($imports -eq $null)
	  {
		throw (new-object NullReferenceException -ArgumentList "Changes is null.  Check that the changes file has data.")
	  }
	Write-Host "Importing changes into T&A environment"
	$undoneImports = $imports | Import-FIMConfig
	if($undoneImports -eq $null)
	  {
		Write-Host "Import complete."
	  }
	else
	  {
		Write-Host
		Write-Host "There were " $undoneImports.Count " uncompleted imports."
		$undoneImports | ConvertFrom-FIMResource -file $undoneFile
		Write-Host
		Write-Host "Please see the documentation on how to resolve the issues."
	  }
}

function SyncPortalConfig($sourceFile, $destFile, $tempFile)
{
	$joinrules = @{
		# === Customer-dependent join rules ===
		# Person and Group objects are not configuration will not be migrated.
		# However, some configuration objects like Sets may refer to these objects.
		# For this reason, we need to know how to join Person objects between
		# systems so that configuration objects have the same semantic meaning.
		Person = "AccountName";
		Group = "DisplayName";

		# === Policy configuration ===
		# Sets, MPRs, Workflow Definitions, and so on. are best identified by DisplayName
		# DisplayName is set as the default join criteria and applied to all object
		# types not listed here.
		#"ma-data" = "Description";

		# === Schema configuration ===
		# This is based on the system names of attributes and objects
		# Notice that BindingDescription is joined using its reference attributes.
		ObjectTypeDescription = "Name";
		AttributeTypeDescription = "Name";
		BindingDescription = "BoundObjectType BoundAttributeType";

		# === Portal configuration ===
		ConstantSpecifier = "BoundObjectType BoundAttributeType ConstantValueKey";
		SearchScopeConfiguration = "DisplayName SearchScopeResultObjectType Order";
		ObjectVisualizationConfiguration = "DisplayName AppliesToCreate AppliesToEdit AppliesToView"
	}

	$destination = ConvertTo-FIMResource -file $destFile
	if($destination -eq $null)
		{ throw (new-object NullReferenceException -ArgumentList "destination Schema is null.  Check that the destination file has data.") }

	Write-Host "Loaded destination file: " $destFile " with " $destination.Count " objects."

	$source = ConvertTo-FIMResource -file $sourceFile
	if($source -eq $null)
		{ throw (new-object NullReferenceException -ArgumentList "source Schema is null.  Check that the source file has data.") }

	Write-Host "Loaded source file: " $sourceFile " with " $source.Count " objects."
	Write-Host
	Write-Host "Executing join between source and destination."
	Write-Host
	$matches = Join-FIMConfig -source $source -target $destination -join $joinrules -defaultJoin DisplayName
	if($matches -eq $null)
		{ throw (new-object NullReferenceException -ArgumentList "Matches is null.  Check that the join succeeded and join criteria is correct for your environment.") }
	Write-Host "Executing compare between matched objects in source and destination."
	$changes = $matches | Compare-FIMConfig
	if($changes -eq $null)
		{ throw (new-object NullReferenceException -ArgumentList "Changes is null.  Check that no errors occurred while generating changes.") }
	Write-Host "Identified " $changes.Count " changes to apply to destination."
	$changes | ConvertFrom-FIMResource -file $tempFile
	Write-Host "Sync complete."
}

$functions = {
	function GetPortalConfig($filename, $serverFQDN, $creds, $AllLocales)
	{
		if(@(get-pssnapin | where-object {$_.Name -eq "FIMAutomation"} ).count -eq 0) {add-pssnapin FIMAutomation}
		$uri="http://" + $serverFQDN + ":5725/ResourceManagementService"
		if ($AllLocales -eq $true)
			{ $policy = Export-FIMConfig -uri $uri -credential $creds -policyConfig -portalConfig -MessageSize 9999999 –AllLocales }
		else
			{ $policy = Export-FIMConfig -uri $uri -credential $creds -policyConfig -portalConfig -MessageSize 9999999 }
		$policy | ConvertFrom-FIMResource -file $filename
	}
}

#### Main Script ####
$creds=Get-Credential -message "Enter credentials for source FIM"
$myargs=@($sourceFileName, $sourceServer, $creds, $AllLocales.IsPresent)
start-job -name "SourceFIM" -init $functions -script { GetPortalConfig $args[0] $args[1] $args[2] $args[3] } -ArgumentList $myargs

$creds=Get-Credential -message "Enter credentials for destination FIM"
$myargs=@($destFileName, $destServer, $creds, $AllLocales.IsPresent)
start-job -name "DestFIM" -init $functions -script { GetPortalConfig $args[0] $args[1] $args[2] $args[3] } -ArgumentList $myargs

write-host "Waiting for Policy Export to complete..."
get-job | wait-job

Write-Host "Exports complete: Starting Policy compare..."
Write-Host
SyncPortalConfig $sourceFilename $destFilename $tempDeltaFile

Write-Host "`nReplace configuration data with destination values"
$portalData=(get-content $tempDeltaFile)

foreach ($ReplaceConfig in $config.DeployConfig.ReplaceData)
{
	$newPortalData=@()
	foreach ($CurrentLine in $portalData)
	{
		$newPortalData+=$CurrentLine.replace($ReplaceConfig.SearchString, $ReplaceConfig.ReplaceString)
	}
	$portalData=$newPortalData
}

$portalData | set-content $finalDeltaFile

Write-Host "`nRemoving excluded Objects from deploy data"
$deployXML=[XML](get-content $finalDeltaFile)

foreach ($ReplaceConfig in $config.DeployConfig.IgnoreData)
{
	$deleteObjects=$deployXML.Results.ImportObject | where { $_.TargetObjectIdentifier -eq "urn:uuid:"+$ReplaceConfig.SearchGUID }
	foreach ($delObj in $deleteObjects)
	{
		write-host "Deleting " $delObj.ObjectType ": " $delObj.AnchorPairs.JoinPair.AttributeValue
		try
		{
			$deployXML.Results.RemoveChild($delObj) | out-null
		}
		catch [System.Management.Automation.MethodInvocationException]
		{
			write-host "Ignoring " $delObj.ObjectType ": " $delObj.AnchorPairs.JoinPair.AttributeValue " not in DeploymentData"
		}
	}
}

$deployXML.Save($finalDeltaFile)
Write-Host "`n`nFinal file for deployment created: " $finalDeltaFile

$input=Read-Host "Do you want commit changes to destination FIM ? (y/n)"
if ($input -eq "y")
{
	CommitPortalChanges $finalDeltaFile
}

This is the configuration XML file: In IgnoreData the Name is only for viewing what object this is, all search is done on the GUID, which is the GUID of the target system.

<?xml version="1.0" encoding="utf-8"?>
<DeployConfig>
	<ReplaceData Name="NetbiosDomain">
		<SearchString>SOURCE-DOM</SearchString>
		<ReplaceString>TARGET-DOM</ReplaceString>
	</ReplaceData>
	<ReplaceData Name="DomainSID">
		<SearchString>S-1-5-21-xxxxxxxxx-xxxxxxxxxx-xxxxxxxxx</SearchString>
		<ReplaceString>S-1-5-21-xxxxxxxxxx-xxxxxxxxx-xxxxxxxxx</ReplaceString>
	</ReplaceData>
	<ReplaceData Name="LDAPDomain1">
		<SearchString>DC=source,DC=com</SearchString>
		<ReplaceString>DC=target,DC=com</ReplaceString>
	</ReplaceData>
	<IgnoreData Name="Set: All GroupAdmins">
		<SearchGUID>95833928-23e0-4e3d-bcc0-b824f3a8e123</SearchGUID>
	</IgnoreData>
	<IgnoreData Name="Set: All UserAdmin">
		<SearchGUID>28279e59-14b7-4b89-9d8a-b4c666a65301</SearchGUID>
	</IgnoreData>
</DeployConfig>

Due to the implementation of PowerShell jobs, if you get any error or exceptions you can retrieve the output from both background jobs with the following command:

Receive-Job

Or

Receive-Job –name

The script also creates the undone.xml file like the original join script, so you can use the original ResumeUndoneImport.ps1 script to retry importing the unprocessed changes.
In addition to this I have built a little helper script that would generate the XML for the objects I don’t want to be deployed and copy this directly to the clipboard, so you can easily add a lot of objects in a short time.

PARAM([string]$DisplayName)
set-variable -name URI -value "http://localhost:5725/resourcemanagementservice" -option constant

Write-Host "- Reading Set Information from Portal"
if(@(get-pssnapin | where-object {$_.Name -eq "FIMAutomation"} ).count -eq 0)
{add-pssnapin FIMAutomation}

$exportObject = export-fimconfig -uri $URI `
                               -onlyBaseResources `
                               -customconfig ("/Set[DisplayName='$DisplayName']")
if($exportObject -eq $null) {throw "Cannot find a set by that name"}
$ResourceID = $exportObject.ResourceManagementObject.ResourceManagementAttributes | `
                Where-Object {$_.AttributeName -eq "ObjectID"}

Write-Host "DisplayName: $DisplayName has ObjectID: " $ResourceID.Value
Write-Host
Write-Host "The following output is added to your clipboard:"
Write-Host
$objectGUID=$ResourceID.Value.Replace("urn:uuid:","")
$output=@()
$output+="`t"
$output+="`t`t$ObjectGUID"
$output+="`t"
$output | clip
$output
Write-Host

Notes:

This script has currently one small issue, each of the replacement objects are currently deployed every time you use this script, as the delta calculation see differences between the environments, and replacement is done after sync and calculate, but that’s ok it still works fine.

I will do an update of the script in maybe the near future to check if the replacement is already done to avoid this, but I haven’t find a way to do so right now.