Overview
The MungeFileHandler allows you to modify the contents of a file. The table below shows the destinations currently supported by the handler along with the expected format of files / directories in the plan.
Destination | File Format | Directory Format |
---|---|---|
Windows Server | C:\Temp\MyFile.txt | C:\Temp\MyDirectory\ |
Network Attached Storage (NAS) | \\server\share$\MyDir\MyFile.txt | \\server\share$\MyDir\ |
Amazon S3 Buckets | s3://bucket/mydir/myfile001.txt | s3://bucket/mydir/ |
Note: Directories must ALWAYS end in a slash. This is mainly because in Windows, there's no way to tell, from the naming convention, if an object is a directory or a file with no file extension.
Plan Details
Config
The config section of the plan specifies what technique should be used to modify the file contents, as well as directives on how the file(s) should be treated while being transformed. It also contains configuration sections to connect to network/cloud based endpoints that require authentication and additional configuration.
Sample
Handler:
Type: Synapse.Handlers.FileUtil:MungeFileHandler
Config:
Type: Yaml
Values:
Type : INI
BackupSource: false
CreateSettingIfNotFound: true
StopOnError: true
RunSequential: true
Aws:
AccessKey: xxxxxxxx
SecretKey: xxxxxxxx
Region: eu-west-1
Element | Type/Value | Required | Description |
---|---|---|---|
Type | "Regex" "XmlTransform" "XPath" "KeyValue" "INI" |
Yes | Indicates how the file will be transformed. Details of each type of transformation can be found here |
BackupSource | Boolean | No | Creates a backup of the original file. The backup file will have the same name as the original file with a timestamp appened. If "StopOnError" is false, failure to backup the source file will NOT stop the attempt to modify the file. (Default = false) |
CreateSettingIfNotFound | Boolean | No | Creates a new setting in the destination file if the source file does not contain the setting already. (Default = false) |
StopOnError | Boolean | No | Should Action Continue When An Error Is Encountered. (Default = true) |
RunSequential | Boolean | No | Runs the files in the order they appear in the plan. This is useful for chained dependencies between file modifications. (Default = false) |
Aws | AwsConfig | No* | Details on how to connect to Amazon Web Services to access S3 Buckets. * = Required if any endpoint is an S3 bucket. |
AwsConfig
This configuration section contains information on how to connect to Amazon Web Services to access the S3 endpoints. This is a required section if any endpoint is an S3 bucket. For optional elements, the values will be pulled from the environment variables of the server and/or the AWS credentails file on the server.
Element | Type/Value | Required | Description |
---|---|---|---|
AccessKey | String | No | The Access Id Key used to make programatic requests to AWS. |
SecretKey | String | No | The Secret Access Key used to make programatic requests to AWS. |
Region | String | Yes | A string value representing a valid Amazon RegionEndpoint. While strictly speaking, AWS S3 doesn't require a Region Endpoint, the client being used to access S3 needs to connect somewhere, so the endpoint is required. |
Parameters
The Parameters section is a list of files to be transformed. If no destination is provided, the transformation will happen "in place" (overwrite the original source file). The "SettingsFile" and "Settings" section explain how the file is to be transformed, and is specific to each type of transformation. More details on what values are appropriate there can be found here.
Sample
Parameters:
Type: Yaml
Values:
Files:
- Source: C:\Temp\FileMunge\input.ini
Destination: s3://mybucket/Temp/FileMunge/output/output.ini
Type: INI
SettingsFile:
Name: \\localhost\c$\Temp\FileMunge\initransform.csv
HasEncryptedValues: true
CreateSettingsIfNotFound: false
Settings:
- Key: TESTKEY
Value: Guy Waguespack Was Here
- Key: "[ODBC 32 bit Data Sources]:TESTKEY"
Value: Guy Waguespack Was Here Too
Element | Type/Value | Required | Description |
---|---|---|---|
Source | String | Yes | The source file to be transformed. |
Destination | String | No | The destination for the transformed file. If no value provided, the transformation will overwrite the source file (in-place transformation). |
Type | "Regex" "XmlTransform" "XPath" "KeyValue" "INI" |
Yes | Overrides the default "Type" specified in the config section. Applies ONLY to this file. Details of each type of transformation can be found here |
CreateSettingIfNotFound | Boolean | No | Same as the config-element of the same name, but applies only to this particular file transformation. This is used to "override" the plan-level setting. |
Settings | List of Key-Value Pairs | No | This is a list of settings that are to be applied to the transformation process. These values are applied AFTER the SettingsFile has been applied, and are executed in the order they appear in the list. |
SettingsFile | |||
-Name | String | No | A file containing transformation information, either in CSV or XML format. See individual transformations for more details. |
-HasEncryptedValues | Boolean | No | Tells the handler that the file contains one or more encrypted values, so attempt to decrypt all values using the Crypto information provided. (Default = false) |
-Crypto | CryptoProvider | No | Contains the information needed to decrypt any values in the file. These values will override any Crypto details provided at the plan level and are only necessary here if the details used to encrypt the values are different from the plan level cryptography. A more detailed description on plan cryptography can be found here. |
Transformation Types
Regex
Performs a line-by-line Regular expression search, replacing any "Key" found with the "Value" specified. Regex grouping can be used to save sections from the original match string and apply them into the replacement value. Below are examples of how group matching works.
Simple Regular-Expression
Performs normal Regex replacement since no match groups were specified.
Key: ABCDEFGHI
Value: xxx
Original: ABCDEFGHI
Result: xxx
Save All Groups As Is
Auto-saves all "groups" since no individual group match variables were specified in the value.
Key: (ABC)DEF(GHI)
Value: xxx
Original: ABCDEFGHI
Result: ABCxxxGHI
Save Select Groups
Saves only first group (ABC) and discards (GHI) since group match varables are used in the value.
Key: (ABC)DEF(GHI)
Value: ${1}xxx
Original: ABCDEFGHI
Result: ABCxxx
SettingsFile Format
Comma-seperated values (CSV) with the Regex match string as the first value and replacement value as the 2nd.
(OLFCMD_URL\s*=\s*).*?$ , http://mynewurl.com
(ENDUR_JRE\s*=\s*).*?$ , ${1}jdk1.9.0.69
Settings
The "Key" values is the Regex match string, while the "Value" value is the value to replace it with.
Settings:
- Key: (SET\s*ENDUR_VER\s*=\s*).*?$
Value: ${1}NEW_VERSION_NUMBER_HERE
XmlTransform
Uses an XmlTransformableDocument (Microsoft.Web.XmlTransform) to apply changes to an Xml Document. Details on this process can be found at CodePlex.
SettingsFile Format
An Xml document representing the values to be transformed. The format for this document can be found at CodePlex.
Settings
Individual settings do not apply to XmlTransformation.
XPath
Uses X-path based parsing of an XML document to replace values, with the "Key" being an X-Path formatted string, and the "Value" being the value to replace.
SettingsFile Format
Comma-seperated values (CSV) where the first value is the X-Path location for the value to be replaces (Key) and the second value is the value to replace (Value).
"//configuration/applicationSettings/MyService.Properties.Settings/setting[@name="initVector"]/value", "MyNewValue"
"//configuration/applicationSettings/MyService.Properties.Settings/setting[@name="KeyContainerName"]/value", "MyOtherNewValue"
Settings
The "Key" contains the X-Path location for the value to be replaces, the "Value" contains the value to replace it with.
Settings:
- Key: '//configuration/applicationSettings/MyService.Properties.Settings/setting[@name="passPhrase"]/value'
Value: SomePassPhrase
- Key: '//configuration/applicationSettings/MyService.Properties.Settings/setting[@name="saltValue"]/value'
Value: SomeSaltyValue
KeyValue
Transforms a simple Key/Value based file where key and value are seperated by a colon or equal sign on the same line.
SettingsFile Format
Comma-seperated values (CSV) where the first value is the "Key" to find, and the second value is the "Value" to replace the existing value with.
"baseGroup.ABCDEFG-svrMessageLogger.component", "New Value From File"
"MyKey", "MyNewValue"
Settings
The "Key" value is the key (value before the first ":" or "=" on the line) to find, while the "Value" is the value to replace the existing value with.
Settings:
- Key: baseGroup.ABCDEFG-svrMessageLogger.isLogging
Value: false
INI
An INI file is treated as a Key/Value property file with the addition of optional "Sections". A key "MyKey" in "Section1" is a different property than the key "MyKey" in "Section2", or "MyKey" in no section at all. The replacement of values is the same as the KeyValue transformation, only the method of finding the Key/Value pair to replace can be slight different if the value is under a "section".
Note: If the Key contains an "=" or ":" in it, the values must be escaped with a slash. Also, shame on you for designing such a bad property file. :)
SettingsFile Format
Comma-seperated values (CSV) where
- If the line contains 2 values, the first is the "Key" and the second is the "Value".
- If the line contains 3 values, the first is the "Section", the second is the "Key" and the third is the "Value".
"SECTION2", "NewKeyFromFile", "New Value From File"
"NewKeyFromFile", "New Value From File"
"SECTION2", "This is a key, not a section"
"key\=key", "guy\=guy"
Settings
Same as with Key/Value transformation, but if you need to specify a "Section", it should be pre-pended to the "Key" and seperated with a colon. (See example below)
Settings:
- Key: TESTKEY
Value: Guy Waguespack Was Here
- Key: "[ODBC 32 bit Data Sources]:TESTKEY"
Value: Guy Waguespack Was Here Too