´óÊý¾ÝµÄ´¦ÀíÔõôÄܱä¿ìÒ»µã£¬´ð°¸ÊÇÇëÓÃspark£¬ÒòΪËüÊÇ»ùÓÚÄÚ´æµÄ£¬¿ÉÒÔÓÐЧ¼õÉÙÊý¾ÝµÄÂ䵨´ÎÊý¡£SparkÐÔÄܳ¬¹ýHadoop°Ù±¶£¬´Ó¶àµü´úÅúÁ¿´¦Àí³ö·¢£¬¼æÊÕ²¢ÐîÊý¾Ý²Ö¿â¡¢Á÷´¦ÀíºÍͼ¼ÆËãµÈ¶àÖÖ¼ÆË㷶ʽ£¬ÊǺ±¼ûµÄÈ«ÄÜÑ¡ÊÖ¡£
Spark²ÉÓÃÒ»¸öͳһµÄ¼¼Êõ¶ÑÕ»½â¾öÁËÔÆ¼ÆËã´óÊý¾ÝµÄÈçÁ÷´¦Àí¡¢Í¼¼¼Êõ¡¢»úÆ÷ѧϰ¡¢NoSQL²éѯµÈ·½ÃæµÄËùÓкËÐÄÎÊÌ⣬¾ßÓÐÍêÉÆµÄÉú̬ϵͳ£¬ÕâÖ±½Óµì¶¨ÁËÆäÒ»Í³ÔÆ¼ÆËã´óÊý¾ÝÁìÓòµÄ°ÔÖ÷µØÎ»¡£
°éËæSpark¼¼ÊõµÄÆÕ¼°Íƹ㣬¶ÔרҵÈ˲ŵÄÐèÇóÈÕÒæÔö¼Ó¡£SparkרҵÈ˲ÅÔÚδÀ´Ò²ÊÇÖËÊÖ¿ÉÈÈ£¬Çá¶øÒ×¾Ù¿ÉÒÔÄõ½°ÙÍòµÄн³ê¡£¶øÒªÏë³ÉΪSpark¸ßÊÖ£¬Ò²ÐèÒªÒ»ÕÐһʽ£¬´ÓÄÚ¹¦Á·Æð£ºÍ¨³£À´½²ÐèÒª¾ÀúÒÔϼ¸¸ö½×¶Î£º
Ò»£ºÊìÁ·µÄÕÆÎÕScalaÓïÑÔ
Spark¿ò¼ÜÊDzÉÓÃScalaÓïÑÔ±àдµÄ£¬¾«Ö¶øÓÅÑÅ¡£ÒªÏë³ÉΪSpark¸ßÊÖ£¬Äã¾Í±ØÐëÔĶÁSparkµÄÔ´´úÂ룬¾Í±ØÐëÕÆÎÕScala¡£
ËäȻ˵ÏÖÔÚµÄSpark¿ÉÒÔ²ÉÓöàÓïÑÔJava¡¢PythonµÈ½øÐÐÓ¦ÓóÌÐò¿ª·¢£¬µ«ÊÇ×î¿ìËٵĺÍÖ§³Ö×îºÃµÄ¿ª·¢APIÒÀÈ»²¢½«ÓÀÔ¶ÊÇScala·½Ê½µÄAPI£¬ËùÒÔÄã±ØÐëÕÆÎÕScalaÀ´±àд¸´Ôӵĺ͸ßÐÔÄܵÄSpark·Ö²¼Ê½³ÌÐò
ÓÈÆäÒªÊìÁ·ÕÆÎÕScalaµÄtrait¡¢apply¡¢º¯Êýʽ±à³Ì¡¢·ºÐÍ¡¢Äæ±äÓëбäµÈ;
¶þ£º¾«Í¨Sparkƽ̨±¾ÉíÌṩ¸ø¿ª·¢ÕßAPI
ÕÆÎÕSparkÖÐÃæÏòRDDµÄ¿ª·¢Ä£Ê½£¬ÕÆÎÕ¸÷ÖÖtransformationºÍactionº¯ÊýµÄʹÓÃ;ÕÆÎÕSparkÖеĿíÒÀÀµºÍÕÒÀÀµÒÔ¼°lineage»úÖÆ;
ÕÆÎÕRDDµÄ¼ÆËãÁ÷³Ì£¬ÀýÈçStageµÄ»®·Ö¡¢SparkÓ¦ÓóÌÐòÌá½»¸ø¼¯ÈºµÄ»ù±¾¹ý³ÌºÍWorker½Úµã»ù´¡µÄ¹¤×÷ÔÀíµÈ
Èý£ºÉîÈëSparkÄÚºË
´Ë½×¶ÎÖ÷ÒªÊÇͨ¹ýSpark¿ò¼ÜµÄÔ´ÂëÑжÁÀ´ÉîÈëSparkÄں˲¿·Ö£º
ͨ¹ýÔ´ÂëÕÆÎÕSparkµÄÈÎÎñÌá½»¹ý³Ì;ͨ¹ýÔ´ÂëÕÆÎÕSpark¼¯ÈºµÄÈÎÎñµ÷¶È;ÓÈÆäÒª¾«Í¨DAGScheduler¡¢TaskSchedulerºÍWorker½ÚµãÄÚ²¿µÄ¹¤×÷µÄÿһ²½µÄϸ½Ú;
ËÄ£ºÕÆÎÕ»ùÓÚSparkÉϵĺËÐÄ¿ò¼ÜµÄʹÓÃ
Spark×÷ÎªÔÆ¼ÆËã´óÊý¾Ýʱ´úµÄ¼¯´ó³ÉÕߣ¬ÔÚʵʱÁ÷´¦Àí¡¢Í¼¼¼Êõ¡¢»úÆ÷ѧϰ¡¢NoSQL²éѯµÈ·½Ãæ¾ßÓÐÏÔÖøµÄÓÅÊÆ£¬ÎÒÃÇʹÓÃSparkµÄʱºò´ó²¿·Öʱ¼ä¶¼ÊÇÔÚʹÓÃÆäÉϵĿò¼ÜÀýÈçShark¡¢Spark StreamingµÈ.
Spark StreamingÊǷdz£³öÉ«µÄʵʱÁ÷´¦Àí¿ò¼Ü£¬ÒªÕÆÎÕÆäDStream¡¢transformationºÍcheckpointµÈ;
SparkµÄÀëÏßͳ¼Æ·ÖÎö¹¦ÄÜ£¬Spark 1.0.0°æ±¾ÔÚSharkµÄ»ù´¡ÉÏÍÆ³öÁËSpark SQL£¬ÀëÏßͳ¼Æ·ÖÎöµÄ¹¦ÄܵÄЧÂÊÓÐÏÔÖøµÄÌáÉý£¬ÐèÒªÖØµãÕÆÎÕ;
Î壺×öÉÌÒµ¼¶±ðµÄSparkÏîÄ¿
ͨ¹ýÒ»¸öÍêÕûµÄ¾ßÓдú±íÐÔµÄSparkÏîÄ¿À´¹á´©SparkµÄ·½·½ÃæÃ棬°üÀ¨ÏîÄ¿µÄ¼Ü¹¹Éè¼Æ¡¢Óõ½µÄ¼¼ÊõµÄÆÊÎö¡¢¿ª·¢ÊµÏÖ¡¢ÔËάµÈ£¬ÍêÕûÕÆÎÕÆäÖеÄÿһ¸ö½×¶ÎºÍϸ½Ú£¬ÕâÑù¾Í¿ÉÒÔÈÃÄúÒÔºó¿ÉÒÔ´ÓÈÝÃæ¶Ô¾ø´ó¶àÊýSparkÏîÄ¿¡£
´óÊý¾Ý¸É»õ | ÂÛSpark¸ßÊÖÊÇÔõÑùÁ¶³ÉµÄ
Áù£ºÌṩSpark½â¾ö·½°¸
³¹µ×ÕÆÎÕSpark¿ò¼ÜÔ´ÂëµÄÿһ¸öϸ½Ú;¸ù¾Ý²»Í¬µÄÒµÎñ³¡¾°µÄÐèÒªÌṩSparkÔÚ²»Í¬³¡¾°µÄϵĽâ¾ö·½°¸;¸ù¾Ýʵ¼ÊÐèÒª£¬ÔÚSpark¿ò¼Ü»ù´¡ÉϽøÐжþ´Î¿ª·¢£¬´òÔì×Ô¼ºµÄSpark¿ò¼Ü;
Ç°ÃæËùÊöµÄ³ÉΪSpark¸ßÊÖµÄÁù¸ö½×¶ÎÖеĵÚÒ»ºÍµÚ¶þ¸ö½×¶Î¿ÉÒÔͨ¹ý×ÔѧÖð²½Íê³É£¬ËæºóµÄÈý¸ö½×¶Î×îºÃÊÇÓɸßÊÖ»òÕßר¼ÒµÄÖ¸ÒýÏÂÒ»²½²½Íê³É£¬×îºóÒ»¸ö½×¶Î£¬»ù±¾ÉϾÍÊǵ½”ÎÞÕÐʤÓÐÕДµÄʱÆÚ£¬ºÜ¶à¶«Î÷ÒªÓÃÐÄÁìÎò²ÅÄÜÍê³É¡£