i have network of industrial sensors, sensors upload csv files ubuntu instance in google compute engine, recieve @ least 10 files each minute, files have between 30 - 2000 lines several values.
i have python script processing files running each minute, script isn't able process files whithin minute, send info mysql db (i need update dashboard every minute) run script in several instances, each 1 processing files. (i have mysql table keep control of wich file being processed or hasn't been processed yet)
so how share disk csv file being stored between several instances without need run rsync?
also, don't see using google cloud storage option, because i'd requiere upload files gcs sync folder in each instance using gscloud command line tool.
i still advice use cloud storage intermediate , use object change notifications. gcs starting sending push notifications file arrived. if script fails, exponential backoff.
run python script on appengine handle push notifications automatically scale frequency of incoming files.
your question regarding shared disk : can mount disk on 16 instances @ once if mounts readonly. can mount 1 gce instance when mounted readwrite. need copy data around, no way around it.
Comments
Post a Comment