Study on IoT systems design

Nota:

• Get prerequiments for the system design. 
• Kurs: A developer's guide to Exploring and Visualizing IoT Datahttps://www.coursera.org/learn/exploring-visualizing-iot-data
• Artykuł: IoT 101 – Everything You Need to Know to Start Your IoT Project http://www.datasciencecentral.com/profiles/blogs/iot-101-everything-you-need-to-know-to-start-your-iot-project
• Koniecznie!!! https://github.com/onurakpolat/awesome-bigdata#data-visualization
• IoT architecture on the edge http://www.iotcentral.io/blog/the-iot-architecture-at-the-edge?context=category-Platforms
• Artykuły do projektu związanego z IoT http://www.datasciencecentral.com/profiles/blogs/16-great-iot-articles-published-in-2016

1. Types of data analysis
   1. Real Time Analysis
      1. Stream Processing
         1. Apache Storm
         2. Apache Spark Streams

            Nota:

            • Mastering Apache Spark 2 https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/
         3. Apache Samza
         4. MOA Framework

            Nota:

            • http://moa.cms.waikato.ac.nz/
   2. Batch Analysis
      1. Apache Hadoop
2. Database Systems
   1. DBMS

      Nota:

      • Database management system
      1. Apache HBase
      2. Apache CouchDB

         Nota:

         • http://db-engines.com/en/system/Cloudant%3BCouchDB%3BCouchbase
         1. IBM Cloudant NoSql
         2. Couchbase
   2. TSDB

      Nota:

      • (Time Series DataBases)
      1. OpenTSDB

         Nota:

         • Data store using HBase
         • Brain Monitoring with Kafka, OpenTSDB, and Grafana http://www.kdnuggets.com/2016/08/brain-monitoring-kafka-opentsb-grafana.html/2
3. File Systems
   1. HDFS

      Nota:

      • Hadoop File System
4. Messaging Queue Systems
   1. Apache Kafka

      Nota:

      • Kafka for beginners https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html
      • Znaczenia Kafka dla IoT https://thenewstack.io/apache-kafka-cornerstone-iot-data-platform/
5. Programming Languages
   1. R Lang
      1. R Open

         Nota:

         • Data visualisation with Shiny https://www.ibm.com/developerworks/community/files/basic/anonymous/api/library/1b6a2624-dc86-4856-b4ed-cdda6bfdecda/document/d26301ed-3155-4e21-ba3d-4936e0bd45cb/media
      2. MS R Server

         Nota:

         • Płatny
      3. Data Science with R

         Nota:

         • Dyplr tricks https://www.r-bloggers.com/lesser-known-dplyr-tricks/
         1. Data Import
            1. Amelia - uzupełnianie danych TS
            2. tidyr - porządkowanie danych
            3. readr - ułatwia import danych
            4. tibble - tab. struktura danych
   2. Python
      1. Interactive Parallel Computing in Python

         Nota:

         • Interactive Parallel Computing in Python https://ipyparallel.readthedocs.io/ https://github.com/ipython/ipyparallel
      2. Clustering Alghoritms

         Nota:

         • http://www.kdnuggets.com/2017/03/k-means-clustering-algorithms-intro-python.html?utm_content=buffer4b146&utm_medium=social&utm_source=facebook.com&utm_campaign=buffer
      3. Biblioteki - Python
         1. EIT
            1. pyEIT

               Nota:

               • https://github.com/liubenyuan/pyEIT
               1. Calls Diagram

                  Nota:

                  • C:\Anaconda3\Scripts>python pycallgraph graphviz -o main.png -- D:\tomas\Python\pyEIT\demo\demo_dynamic_greit.py
                  • http://pycallgraph.slowchop.com/en/master/guide/intro.html
                  • Get readoff of Windows error - http://delimitry.blogspot.com/2015/03/solving-problems-with-pycallgraph-under.html
            2. Rpi.EIT

               Nota:

               • https://github.com/agungdwiprasetyo/RPi.EIT-algorithms
         2. Siatki - Mesh
            1. MeshPy

               Nota:

               • Site https://mathema.tician.de/software/meshpy/
            2. Python Triangle

               Nota:

               • Implementacja w Python biblioteki Trianle http://dzhelil.info/triangle/#
               • Biblioteka Triangle: http://www.cs.cmu.edu/~quake/triangle.html
            3. PyDistMesh

               Nota:

               • https://pypi.python.org/pypi/PyDistMesh
         3. Zagad. proste i odwrotne
            1. pyGimli

               Nota:

               • http://www.pygimli.org/index.html
               • Anaconda install: https://anaconda.org/gimli/pygimli
            2. Operator Discretization Library

               Nota:

               • https://odlgroup.github.io/odl/getting_started/about_odl.html
            3. Algorithmic Differentiation in Python

               Nota:

               • http://pythonhosted.org/algopy/
            4. Linear Inverse Problem
               1. linvpy

                  Nota:

                  • https://github.com/LCAV/linvpy
               2. pinvprob

                  Nota:

                  • https://github.com/HajimeKawahara/pinvprob
               3. hIPPYlib

                  Nota:

                  • https://hippylib.github.io/tutorials/2_PoissonDeterministic/
         4. Math Lib.
            1. Numpy

               Nota:

               • Kurs Numpy http://www.python-course.eu/numpy.php
            2. SciPy

               Nota:

               • Wykaz oprogramowania naukowego https://www.scipy.org/topical-software.html
               • SciPy dokumentacja https://www.scipy.org/scipylib/index.html
               1. Voronoi Diagrams

                  Nota:

                  • http://zderadicka.eu/voronoi-diagrams/
                  • http://www-astro.physics.ox.ac.uk/~mxc/software/
                  • http://dzhelil.info/triangle/examples.html
         5. ERT
            1. BERT

               Nota:

               • Repo https://gitlab.com/resistivity-net/bert/tree/master
            2. E4D

               Nota:

               • https://e4d.pnnl.gov/Documents/E4D_User_Guide.pdf
         6. MISC
            1. TomograPy

               Nota:

               • Strona projektu http://nbarbey.github.io/TomograPy/
               • Artykuł: https://arxiv.org/pdf/1103.5904.pdf
            2. Medical Data Manipulation

               Nota:

               • https://idoimaging.com/programs?q%5Bdisplay_function%5D=&q%5Bfor_audience%5D=&q%5Bfunction%5D=&q%5Bheader_function%5D=&q%5Binterface%5D=&q%5Blanguage%5D=19&q%5Bname_cont%5D=&q%5Bnetwork_function%5D=&q%5Bother_function%5D=&q%5Bplatform%5D=&q%5Bprogramming_function%5D=&q%5Bread_format%5D=&q%5Bspeciality%5D=&q%5Bsummary_cont%5D=&q%5Bwrite_format%5D=
         7. Data Flow
            1. Apache AirFlow

               Nota:

               • Tylko Linux http://airflow.incubator.apache.org/start.html
         8. Visualization
            1. VTK

               Nota:

               • http://www.vtk.org/Wiki/VTK/Examples/Python
            2. Matplotlib
      4. Intel Python

         Nota:

         • Installation: https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda
      5. Python (x,y)

         Nota:

         • https://python-xy.github.io/
   3. Scala

      Nota:

      • Development and deployment of Spark applications with Scala, Eclipse, and sbt – Part 1: Installation & configuration http://www.nodalpoint.com/development-and-deployment-of-spark-applications-with-scala-eclipse-and-sbt-part-1-installation-configuration/
      • Visualizing Big Data with Spark and Scala http://blog.ibmjstart.net/2016/07/18/visualizing-big-data-spark-scala/
      1. SBT
      2. Scala IDE

         Nota:

         • http://scala-ide.org/download/sdk.html
   4. Java
   5. Flow Base Programming

      Nota:

      • Used in IBM Streams, Apache NiFi
      • http://www.jpaulmorrison.com/fbp/
      • Apache Beam (Pyton SDK - https://github.com/apache/beam/tree/master/sdks/python#install-virtualenv) Google Cloud DataFlow - środowisko uruchomieniowe dla Apache Beamhttps://cloud.google.com/dataflow/  
      • Opis modelu programistycznego https://beam.apache.org/documentation/programming-guide/   
   6. Programmer IDE
      1. Eclipse
         1. Maven
      2. Zeppelin Notebook

         Nota:

         •  Zbiór przykładów analizy danych do Zeppelina https://github.com/hortonworks-gallery/zeppelin-notebooks   
      3. Jupiter Notebook

         Nota:

         • Data export to HTML https://blog.dominodatalab.com/lesser-known-ways-of-using-notebooks/
         • Visualisation PixieDust: Magic for Your Python Notebook https://developer.ibm.com/clouddataservices/2016/10/11/pixiedust-magic-for-python-notebook/
         • http://notebooks.azure.com
         • https://nbviewer.jupyter.org/
      4. R Studio
      5. Anaconda
      6. VisualStudio PTVS

         Nota:

         • https://github.com/Microsoft/PTVS/
6. Implementation
   1. VM

      Nota:

      • Virtualization
      1. Hyper-V
      2. Docker

         Nota:

         • Wdrożenie Zeppelin https://github.com/dylanmei/docker-zeppelin
         • Wdrożenie Apache Spark https://github.com/gettyimages/docker-spark
      3. VMware
      4. VirtualBox
   2. Cloud Computing
      1. Azure HDInsight
      2. IBM Bluemix IoT

         Nota:

         • Koniecznie!!! Spark Streaming + Watson IoT Platform Integration https://developer.ibm.com/recipes/tutorials/spark-streaming-ibm-watson-iot-platform-integration/
         • Koniecznie: Engage Machine Learning for detecting anomalous behaviors of things https://developer.ibm.com/recipes/tutorials/engage-machine-learning-for-detecting-anomalous-behaviors-of-things/
         • Koniecznie: Timeseries Data Analysis of IoT events by using Jupyter Notebook  https://developer.ibm.com/recipes/tutorials/timeseries-data-analysis-of-iot-events-by-using-jupyter-notebook/
         • Koniecznie!!! Integrating Watson IoT Platform with Message Hub and Apache Spark  https://developer.ibm.com/recipes/tutorials/integrating-watson-iot-platform-with-message-hubkafka/
         • Koniecznie!!! Integracja DB i Spark https://developer.ibm.com/clouddataservices/docs/ibm-data-science-experience/docs/
         • Cleanse and visualize location data using Spark and Bluemix https://www.ibm.com/developerworks/library/ba-1610location-data-spark-bluemix-trs/index.html
         • Apache Spark - using notebooks https://console.ng.bluemix.net/docs/services/AnalyticsforApacheSpark/index-gentopic3.html#genTopProcId4
         • !!! SparkR notebook with Cloudant https://github.com/charles2588/bluemixsparknotebooks/blob/master/R/SparkR_Cloudant_Connection.ipynb
      3. AWS
      4. Databricks
      5. Google Cloud Platform

         Nota:

         • Real-Time Stream Processing for IoT: https://cloud.google.com/solutions/architecture/real-time-stream-processing-iot
         • Cena na 1 Rok GPU in Google: https://cloud.google.com/products/calculator/#id=99152826-9096-43b2-a100-836c36ef4d50
   3. Bare Metal
7. Integrated Platforms
   1. Hortonworks
      1. Data Platform
      2. Data Flow
   2. Cloudera

      Nota:

      • Cloudera StreamSets https://vimeo.com/147671735
   3. IBM Open Platform
8. Data Integration
   1. Apache NiFi
9. Messaging Protocols
   1. MQTT

      Nota:

      • !!! iot-device-samples https://github.com/ibm-messaging/iot-device-samples
10. Programming Methods
    1. Img. Reconstr. Algorithms
       1. Tensor-based

          Nota:

          • Artykuł: http://iopscience.iop.org/article/10.1088/1361-6501/aa58a3/meta
       2. Simulated Annealing Particle Swarm Optimization
       3. Landweber iterative method
       4. Linear back projection (LBP)
       5. Singular value decomposition

          Nota:

          • Implementacja Python: https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.svd.html
       6. Conjugate Gradient method
       7. Maximum Likelihood Estimation Method

          Nota:

          • https://github.com/clarehchao/Spark3DImageReconstruction
       8. Inverse problems

          Nota:

          • Introduction to inverse problems (Spring 2014) http://cc.oulu.fi/~smaenpaa/inv_pk/inv_pk14_eng.pdf
          1. Tikhonov regularization method

             Nota:

             • Implementacja Python https://pypi.python.org/pypi/InverseProblem/1.0
          2. Statistical and Computational InverseProblems

             Nota:

             • https://www.researchgate.net/profile/Ian_Langmore/publication/268250507_Statistical_and_Computational_Inverse_Problems_and_a_little_uncertainty_quantification/links/5654822708aeafc2aabbd811/Statistical-and-Computational-Inverse-Problems-and-a-little-uncertainty-quantification.pdf?origin=publication_detail&ev=pub_int_prw_xdl&msrp=iG4X9V_68LOGbkxNrcZEP4Az6qj7tC0tBRcUQyjL8QYqQT4QoLmsF2FgkN6bhYKWGD5hIwTkn5ZA4aHZ8A8286ns9G-BWrYTAsuNSMv5CNk.bYd_LZtB1khHvYwW5OyIFeUTFjZCQqyRJayOmj_N5AXLMsdZGfQieOSGqKg4N2g4hxqe2aS8SGqs7RTj0lO2mw.wfseq3OvlHxPmYqbEhxOxL040D2b4044fCPINZ0nDdsCbRE9GxeAhwoYCzzzOf64AwlcmWuBBIbKFhDQjoECqQ.jNjix2pbRzy-BWR2vabpp4TbUNEHGKEiBIwCb6q11EYbYkRGdAy-ev4pUz1krhlO9uaouT9_yrSr7F-4hJheJQ
          3. Lectures

             Nota:

             • Lecture1: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture1 Lecture2: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture2 Lecture3&4: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture3 Lecture5: https://koppa.jyu.fi/en/courses/164831/course-literature/lecture5
    2. Machine Intelligence
       1. Neural Networks
       2. Machine Learning

          Nota:

          • A Visual Introduction to Machine Learning http://www.r2d3.us/visual-intro-to-machine-learning-part-1/
          1. Apache Spark MLlib
       3. Deep Learning

          Nota:

          • Deeplearning4j on Spark https://deeplearning4j.org/
          • TensorFlow http://www.tensorflow.org
          • An Overview of Python Deep Learning Frameworks http://www.kdnuggets.com/2017/02/python-deep-learning-frameworks-overview.html
          • TensorFlow on Sparkhttps://github.com/yahoo/TensorFlowOnSpark
          1. Google TensorFlow

             Nota:

             • Porównanie TensorFlow i CNTK https://esciencegroup.com/2016/02/08/tensorflow-meets-microsofts-cntk/
             1. TensorBoard: Visualizing Learning

                Nota:

                • (C:\Anaconda3\envs\TensorFlow) C:\Users\tom\Documents>python -m tensorflow.tensorboard --logdir=./ localhost:6006
          2. Microsoft Cognitive Toolkit (CNTK)

             Nota:

             • Strona projektu: https://www.microsoft.com/en-us/research/product/cognitive-toolkit/
             • Github: https://github.com/Microsoft/CNTK
             • Wiki: https://github.com/Microsoft/CNTK/wiki
11. Approaches to reconstruction acceleration

    Nota:

    • Artykuł: Review of Parallel Computing Techniques for Computed Tomography Image Reconstruction http://www.imaging.sbes.vt.edu/BIDLib/CT/NiLiWang_ParaCompTech.pdf
    • Rozprawa Doktorska dot. CT: http://ir.uiowa.edu/cgi/viewcontent.cgi?article=2330&context=etd
    1. Parallel Computing
       1. Distribiuted Computing
          1. Apache Spark
             1. SparkR Desc.

                Nota:

                • Opis biblioteki SparkR https://spark.apache.org/docs/2.1.0/api/R/index.html
             2. DataFrame
                1. Operations

                   Nota:

                   • Operacje na obiekcie SparkDataFrame https://www.analyticsvidhya.com/blog/2016/10/spark-dataframe-and-operations/
       2. Local PC Clusters
    2. GPU

       Nota:

       • Artykuł: P.Kapusta http://annals-csis.org/Volume_8/pliks/344.pdf
    3. Improved Algorithms
12. Data Visualization
    1. Web App

       Nota:

       • https://www.slideshare.net/BenLaird/real-time-data-viz-with-spark-streaming-kafka-and-d3js
       1. D3.js

          Nota:

          • https://github.com/d3
          1. DC.js

             Nota:

             • https://dc-js.github.io/dc.js/
       2. Socket.io

          Nota:

          • https://github.com/socketio/socket.io
       3. R Shiny

          Nota:

          • Tutorial: new ver. http://shiny.rstudio.com/tutorial/ old http://rstudio.github.io/shiny/tutorial/
       4. Rickshaw TS graphs.

          Nota:

          • Rickshaw is a JavaScript toolkit for creating interactive time series graphs. http://code.shutterstock.com/rickshaw/
          1. RealTime Visualization IBM

             Nota:

             • https://developer.ibm.com/recipes/tutorials/visualizing-your-data/
       5. Bokeh

          Nota:

          • https://bokeh.pydata.org/en/latest/
       6. MPLD3

          Nota:

          • http://mpld3.github.io/
13. Medical Data Analysis
    1. MIRAN Project

       Nota:

       • Projekt analizy danych medycznych za pomocą Python http://ne-scientific.com/miran/ http://ne-scientific.com/miran/
14. Microservices

    Nota:

    • https://youtu.be/-zsKY9p_5R8?t=34m2s https://github.com/miguelgrinberg/microflack_admin
    • https://youtu.be/nrzLdMWTRMM
    • https://12factor.net/
    1. Loadbalancer
       1. Windows
          1. Nginx

             Nota:

             • https://nginx.org/en/download.html
       2. Linux
          1. HAProxy

             Nota:

             • http://www.haproxy.org/
    2. Design patterns
       1. API Gateway

          Nota:

          • http://microservices.io/patterns/apigateway.html
       2. Service Registry

          Nota:

          • http://microservices.io/patterns/service-registry.html
    3. Service Registry
       1. Consul

          Nota:

          • https://www.consul.io/intro/index.html
       2. etcd

          Nota:

          • https://github.com/coreos/etcd
    4. Containers
       1. Docker
    5. Service-to-Service Communication
    6. Logging
       1. Logspout

          Nota:

          • https://github.com/gliderlabs/logspout