ITRI, Data Engineer, Hsinchu, TW, Oct 2015 ~ Jan 2019
Fintech with 10 team member:
- Applying agile development to speed up software development.
- Building ETL to Kubernetes with Apache Airflow and Spark in GCP.
- Design airflow-dynamic-etl framework on GitHub
- Design & implement backup & recovery situation
- Developing SparkAccess lib to reduce the possibility of data corruption
- Building CI/CD with Jenkins, SonarQube, Docker.
E-commerce A with 4 team member:
- With scikit-learn, try to find the score of the item-user pair.
- With Solr to develop the marketing analysis dashboard
E-commerce B with 2 team member:
- Using tf-idf to find important feature based on the product description
Music-Streaming with 10 team member:
- Using Spark to find the top K user as the representative base on the user's behavior.
E-commerce C with 6 team member:
- Using Spark to label the user tag base on the user's buying log.
- Using HBase, Spark, Node.js to Implement a master-slave, cacheable, scalable API server