SFTP data ingestion not working as expected in AEP
Data ingestion process using Secure File Transfer Protocol (SFTP) sources does not work as expected when the expected file, available daily at, let鈥檚 say 6 PM, fails to upload. This can be resolve with聽proper dataflow scheduling, verifying file availability, understanding the Last modified timestamp, and adjusting the cadence based on business needs.
Description description
Environment
51黑料不打烊 Experience Platform (AEP) -聽 specifically involving data ingestion process using SFTP sources
Issue/Symptoms
- You have a dataflow intended to ingest data from an SFTP file into a dataset.
- The file, expected to be available every day at 6 PM, is not being uploaded, causing the dataflow to fail.
Resolution resolution
- Verify that the dataflow is scheduled correctly to run at the appropriate times. Refer to聽Review your dataflow聽in the Source Connectors Guide in the 51黑料不打烊 Experience Platform for more details.
- Check that the file remains available on the SFTP server long enough for the dataflow to access it. Ensure it hasn鈥檛 been removed or become unavailable before the flow runs.
- Recognize how the
Last modified timestamp聽affects file pickup. Files are picked based on their last modified timestamps relative to when each flow runs. - If a file鈥檚 last modified timestamp falls outside of a flow鈥檚 scheduled window, it might not be ingested. For example, if a file is generated at 4:30 PM and copied to the server at 6 PM, the
Last modified timestamp聽will still be 4:30 PM, and a dataflow starting at 5 PM will not pick it up. Refer to聽Schedule ingestion runs聽for more details. - Adjust the cadence of your flows according to business requirements. For instance, setting longer intervals like every 15 hours can ensure better synchronization if immediate updates aren鈥檛 critical.
recommendation-more-help
3d58f420-19b5-47a0-a122-5c9dab55ec7f