One interface, any storage: building a configurable file-sync tool in .NET

Every team accumulates little file-moving chores. Copy last night's exports to the NAS. Push product images to blob storage. Keep a local mirror of scanned invoices. Each one usually starts life as a PowerShell one-liner, grows retry logic and logging, gets cloned for the next chore with slightly different paths — and two years later there are nine scripts, no two alike, and nobody remembers which ones still run.

This article builds the alternative once: a small .NET console tool where a sync job is pure configuration — a source, one or more targets, an optional file filter — and adding a new chore means adding a JSON block, not writing code. The full tool is about six short C# files, because the heavy lifting sits behind one abstraction.

The abstraction that makes it possible

The reason ad-hoc scripts multiply is that every storage backend has a different API: System.IO for disks, Azure.Storage.Blobs for blob containers, SSH.NET for SFTP. The moment copy logic touches any of those directly, it's married to that backend.

Regira.IO.Storage (Apache-2.0) collapses them behind one interface:

public interface IFileService
{
    string Root { get; }
    Task<bool>                Exists(string identifier);
    Task<byte[]?>             GetBytes(string identifier);
    Task<Stream?>             GetStream(string identifier);
    Task<IEnumerable<string>> List(FileSearchObject? so = null);
    Task<string>              Save(string identifier, Stream stream, string? contentType = null);
    Task                      Move(string sourceIdentifier, string targetIdentifier);
    Task                      Delete(string identifier);
    // + identifier/URI helpers
}

The crucial design decision is what an identifier is: a path relative to the service's root — invoices/2024/inv-001.pdf — never an absolute one. The same identifier means "this file" on a local disk rooted at /var/app/storage, in an Azure container, or on an SFTP server. That's the property that makes copying trivial: read an identifier from the source, write the same identifier to the target, and the folder structure comes along for free.

Implementations exist for the local file system (BinaryFileService), Windows network shares with credentials (NetworkFileService), Azure Blob Storage (BinaryBlobService, in Regira.IO.Storage.Azure), SFTP (SftpService, in Regira.IO.Storage.SSH), GitHub repositories, and even ZIP archives. Our tool starts with two and leaves the door open.

Jobs as configuration

Working backwards from the config we want to write:

// appsettings.json
"Sync": [
    {
        "Source": {
            "FileSystem": { "RootFolder": "C:\\Exports\\Photos" },
            "FileSearchObject": { "Extensions": [ ".jpg", ".jpeg", ".png" ] }
        },
        "Targets": [
            { "FileSystem": { "RootFolder": "F:\\Backup\\Photos" } },
            { "Azure": { "ConnectionString": "<secret>", "ContainerName": "photos" } }
        ]
    },
    {
        "Source":  { "FileSystem": { "RootFolder": "C:\\Scans" } },
        "Targets": [ { "FileSystem": { "RootFolder": "\\\\nas\\archive\\scans" } } ]
    }
]

A list of jobs; each job one source and n targets; the source optionally filtered. The options classes mirror it exactly — and here's the trick that keeps the config honest: a StorageOptions block declares one property per supported backend, and whichever one is present decides the implementation:

public class SyncOptions : List<SyncJobOptions>;

public class SyncJobOptions
{
    public SourceStorageOptions Source { get; set; } = null!;
    public ICollection<StorageOptions> Targets { get; set; } = new HashSet<StorageOptions>();
}

public class StorageOptions
{
    public FileSystemOptions? FileSystem { get; set; }
    public AzureOptions? Azure { get; set; }
    // add more storage option implementations here
}

public class SourceStorageOptions : StorageOptions
{
    public FileSearchObject? FileSearchObject { get; set; }
}

Note there's no "Type": "azure" discriminator to keep in sync with an enum — the presence of the Azure key is the discriminator, and FileSystemOptions / AzureOptions / FileSearchObject are the library's own option types, so configuration binding gives us their typed shape for free.

A tiny factory turns an options block into a live service:

public class StorageFileServiceFactory : IStorageFileServiceFactory
{
    public IFileService Create(StorageOptions options)
    {
        if (options.FileSystem != null)
            return new BinaryFileService(options.FileSystem);
        if (options.Azure != null)
            return new BinaryBlobService(new AzureCommunicator(options.Azure));
        // Add more storage options here
        throw new NotSupportedException();
    }
}

(The IStorageFileServiceFactory, ISyncJobManager and ISyncManager interfaces are the one-method mirrors of the classes that implement them — declared for DI, elided here.)

The sync itself: fifteen lines

With everything storage-specific behind IFileService, the actual synchronization is almost disappointingly small. FileProcessor (a helper from the same package) walks the source — applying the job's FileSearchObject filter, recursively — and invokes a callback per file:

public class SyncJobManager(IStorageFileServiceFactory factory, ILogger<SyncJobManager> logger)
    : ISyncJobManager
{
    public Task Sync(SyncJobOptions options)
    {
        var sourceService = factory.Create(options.Source);
        var fileProcessor = new FileProcessor(sourceService);
        var targetServices = options.Targets.Select(factory.Create);

        return fileProcessor.ProcessFiles(
            options.Source.FileSearchObject ?? new FileSearchObject(),
            async (path, fileService) =>
            {
                var identifier = fileService.GetIdentifier(path);
                foreach (var target in targetServices)
                {
                    if (await target.Exists(identifier)) continue;
                    try
                    {
                        await using var stream = await fileService.GetStream(identifier);
                        await target.Save(identifier, stream!);
                        logger.LogInformation($"Copied {identifier} to {target.Root}");
                    }
                    catch (Exception ex)
                    {
                        logger.LogError(ex, $"Copying {identifier} to {target.Root} failed");
                    }
                }
            }, true);
    }
}

Three details carry more weight than their line count:

• GetIdentifier(path) converts whatever absolute form the walker yields back into the portable relative key before touching the targets. Identifiers in, identifiers out — the discipline that makes source and target interchangeable.
• Exists before Save makes the tool copy-if-missing and therefore idempotent: run it hourly, run it after a crash, run two jobs overlapping the same target — files already present are skipped, so reruns cost roughly one existence check per file. (It also defines what this tool is not: it won't overwrite changed files or delete removed ones. A true mirror would compare timestamps or hashes and is a deliberate step up in destructive potential — a good v2, but not a default.)
• Streams, not byte arrays. GetStream → Save(stream) pipes each file through without ever holding it in memory, which is the difference between "works on my test folder" and "works on 40 GB of video".

The per-file try/catch means one unreadable file logs an error and the run continues — for a nightly batch job, resilience beats fail-fast.

A last one-line class fans out over all configured jobs:

public class SyncManager(IOptions<SyncOptions> options, ISyncJobManager manager) : ISyncManager
{
    public async Task Sync()
    {
        foreach (var job in options.Value)
            await manager.Sync(job);
    }
}

Hosting: the unglamorous parts, done properly

The entry point is a standard Generic Host console — worth having even for a "little tool", because it buys configuration layering, DI and real logging for a dozen lines. The three Add* calls are small extension methods you write yourself in the same file (shown by their intent here — each is a few lines wrapping the standard APIs):

var host = Host.CreateDefaultBuilder(args)
    .AddConfiguration()   // env vars + appsettings.json + user secrets (Debug)
    .AddServices()        // Configure<SyncOptions>(config.GetSection("Sync")) + the services above
    .AddSerilog()         // wraps UseSerilog, reading the "Serilog" config section
    .Build();

using var scope = host.Services.CreateScope();
var sm = scope.ServiceProvider.GetRequiredService<ISyncManager>();
await sm.Sync();

The packages behind it all: Regira.IO.Storage and Regira.IO.Storage.Azure for the storage services, plus the usual hosting suspects — Serilog.Extensions.Hosting, Serilog.Settings.Configuration, Serilog.Sinks.Console, Serilog.Sinks.File and Microsoft.Extensions.Configuration.UserSecrets.

Two habits to keep from the setup — with one precedence trap. Secrets stay out of the committed appsettings.json: user secrets carry the Azure connection string in development, environment variables in production. But mind the provider order your AddConfiguration establishes — in the order shown (env vars first, then JSON), a key present in appsettings.json overrides the environment variable, so the production override only works if the key is absent from the JSON entirely, not placeholdered. (Prefer env-vars-win? Add the JSON provider first.) And Serilog writes to console and a rolling file (logs/FileSync-.log, monthly, capped retention), because a tool like this runs unattended from Task Scheduler or a systemd timer, and the log file is the only witness.

Extending it

The shape of the tool dictates the shape of every future change, which is the real payoff:

• New chore? New JSON block. No build, no deploy.
• New backend? One nullable property on StorageOptions, one branch in the factory. SFTP is SftpService + SftpConfig from Regira.IO.Storage.SSH; a GitHub repo (versioned mirror of a config folder, say) is the GitHub service. The sync loop doesn't change — it never knew about backends to begin with.
• Narrower sources? FileSearchObject already filters by folder, extension list and recursion, so "only PDFs under invoices/" is configuration too.

The whole tool is packaged to run — grab the sample project: the six files, a demo job syncing a bundled data/source folder, and a README with the expected output. Run it twice and watch the second run skip everything.

That's the pattern to take away, independent of any library: put the variation (backends) behind one interface keyed by portable identifiers, put the intent (jobs) in configuration, and the code that remains is a short, uneventful loop you'll never need to clone again.

Package docs: regira.github.io/Regira-Packages — IO.Storage — including the UNC/SFTP/GitHub/ZIP implementations and helpers like ExportHelper and FileNameHelper.